[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240507 08:04:11.120261 9046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.213.190:44625
I20240507 08:04:11.121641 9046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:04:11.122732 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:11.140527 9055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:11.140820 9053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:11.141996 9046 server_base.cc:1034] running on GCE node
W20240507 08:04:11.141713 9052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.257041 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:12.257301 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:12.257515 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069052257488 us; error 0 us; skew 500 ppm
I20240507 08:04:12.258415 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:12.265131 9046 webserver.cc:460] Webserver started at http://127.8.213.190:42021/ using document root <none> and password file <none>
I20240507 08:04:12.266109 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:12.266336 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:12.266790 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:12.273546 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/master-0-root/instance:
uuid: "b2b3a38c57604f748496b35b36ae24e1"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.281507 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240507 08:04:12.286845 9061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:12.288035 9046 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240507 08:04:12.288396 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/master-0-root
uuid: "b2b3a38c57604f748496b35b36ae24e1"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.288775 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:12.346962 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:12.348565 9046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:04:12.349043 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:12.419202 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.190:44625
I20240507 08:04:12.419301 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.190:44625 every 8 connection(s)
I20240507 08:04:12.426748 9113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:12.451156 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1: Bootstrap starting.
I20240507 08:04:12.456871 9113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:12.458851 9113 log.cc:824] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1: Log is configured to *not* fsync() on all Append() calls
I20240507 08:04:12.463647 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1: No bootstrap required, opened a new log
I20240507 08:04:12.487129 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b3a38c57604f748496b35b36ae24e1" member_type: VOTER }
I20240507 08:04:12.487721 9113 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:12.487993 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b3a38c57604f748496b35b36ae24e1, State: Initialized, Role: FOLLOWER
I20240507 08:04:12.488831 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b3a38c57604f748496b35b36ae24e1" member_type: VOTER }
I20240507 08:04:12.489342 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:04:12.489660 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:04:12.490024 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:12.494163 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b3a38c57604f748496b35b36ae24e1" member_type: VOTER }
I20240507 08:04:12.494984 9113 leader_election.cc:302] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2b3a38c57604f748496b35b36ae24e1; no voters:
I20240507 08:04:12.496730 9113 leader_election.cc:288] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 08:04:12.497087 9116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:12.499409 9116 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 1 LEADER]: Becoming Leader. State: Replica: b2b3a38c57604f748496b35b36ae24e1, State: Running, Role: LEADER
I20240507 08:04:12.500463 9116 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b3a38c57604f748496b35b36ae24e1" member_type: VOTER }
I20240507 08:04:12.501502 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:04:12.510967 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2b3a38c57604f748496b35b36ae24e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b3a38c57604f748496b35b36ae24e1" member_type: VOTER } }
I20240507 08:04:12.511739 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [sys.catalog]: This master's current role is: LEADER
I20240507 08:04:12.511518 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2b3a38c57604f748496b35b36ae24e1. Latest consensus state: current_term: 1 leader_uuid: "b2b3a38c57604f748496b35b36ae24e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2b3a38c57604f748496b35b36ae24e1" member_type: VOTER } }
I20240507 08:04:12.512650 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [sys.catalog]: This master's current role is: LEADER
I20240507 08:04:12.515422 9124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:04:12.526474 9124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:04:12.529254 9046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:04:12.544191 9124 catalog_manager.cc:1347] Generated new cluster ID: a363d7eef32b434085f5fbbf22779d86
I20240507 08:04:12.544520 9124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:04:12.568544 9124 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:04:12.570263 9124 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:04:12.585100 9124 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1: Generated new TSK 0
I20240507 08:04:12.586182 9124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:04:12.599582 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:12.609284 9135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:12.610654 9136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.613942 9046 server_base.cc:1034] running on GCE node
W20240507 08:04:12.614478 9138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.615437 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:12.615727 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:12.615943 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069052615926 us; error 0 us; skew 500 ppm
I20240507 08:04:12.616837 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:12.621140 9046 webserver.cc:460] Webserver started at http://127.8.213.129:42663/ using document root <none> and password file <none>
I20240507 08:04:12.621922 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:12.622170 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:12.622532 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:12.624126 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-0-root/instance:
uuid: "865ec2faa51540809fa31393e6a637d6"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.631258 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20240507 08:04:12.636582 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:12.637468 9046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240507 08:04:12.637876 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-0-root
uuid: "865ec2faa51540809fa31393e6a637d6"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.638274 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:12.662971 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:12.664711 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:12.667582 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:04:12.673271 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:04:12.673631 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240507 08:04:12.673960 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:04:12.674188 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:12.764425 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.129:46421
I20240507 08:04:12.764472 9205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.129:46421 every 8 connection(s)
I20240507 08:04:12.772342 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:12.781117 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:12.784027 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.790593 9046 server_base.cc:1034] running on GCE node
W20240507 08:04:12.793349 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.794375 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:12.794589 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:12.794766 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069052794747 us; error 0 us; skew 500 ppm
I20240507 08:04:12.795358 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:12.797305 9206 heartbeater.cc:347] Connected to a master server at 127.8.213.190:44625
I20240507 08:04:12.797804 9206 heartbeater.cc:464] Registering TS with master...
I20240507 08:04:12.799130 9206 heartbeater.cc:511] Master 127.8.213.190:44625 requested a full tablet report, sending...
I20240507 08:04:12.801687 9046 webserver.cc:460] Webserver started at http://127.8.213.130:36525/ using document root <none> and password file <none>
I20240507 08:04:12.802278 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:12.802484 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:12.802798 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:12.803077 9078 ts_manager.cc:194] Registered new tserver with Master: 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421)
I20240507 08:04:12.803997 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-1-root/instance:
uuid: "e251508eda9d45e7acaf131d69144930"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.805966 9078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55494
I20240507 08:04:12.811055 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20240507 08:04:12.815429 9218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:12.816288 9046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240507 08:04:12.816632 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-1-root
uuid: "e251508eda9d45e7acaf131d69144930"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.817029 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:12.836877 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:12.838611 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:12.841224 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:04:12.845003 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:04:12.845263 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:12.845533 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:04:12.845767 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240507 08:04:12.932749 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.130:42579
I20240507 08:04:12.932807 9280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.130:42579 every 8 connection(s)
I20240507 08:04:12.939925 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:12.949543 9284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:12.951748 9285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.953194 9281 heartbeater.cc:347] Connected to a master server at 127.8.213.190:44625
I20240507 08:04:12.953568 9281 heartbeater.cc:464] Registering TS with master...
I20240507 08:04:12.954540 9281 heartbeater.cc:511] Master 127.8.213.190:44625 requested a full tablet report, sending...
I20240507 08:04:12.956763 9078 ts_manager.cc:194] Registered new tserver with Master: e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:12.958484 9078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55496
I20240507 08:04:12.964395 9046 server_base.cc:1034] running on GCE node
W20240507 08:04:12.965165 9287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:12.966171 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:12.966379 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:12.966539 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069052966522 us; error 0 us; skew 500 ppm
I20240507 08:04:12.967057 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:12.969660 9046 webserver.cc:460] Webserver started at http://127.8.213.131:44921/ using document root <none> and password file <none>
I20240507 08:04:12.970153 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:12.970338 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:12.970584 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:12.971710 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-2-root/instance:
uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.976333 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240507 08:04:12.979662 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:12.980499 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240507 08:04:12.980777 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-2-root
uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
format_stamp: "Formatted at 2024-05-07 08:04:12 on dist-test-slave-ls56"
I20240507 08:04:12.981074 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1715069050956306-9046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:13.002727 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:13.004012 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:13.006537 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:04:13.009459 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:04:13.009670 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:13.009918 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:04:13.010082 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:13.101305 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.131:37689
I20240507 08:04:13.101397 9354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.131:37689 every 8 connection(s)
I20240507 08:04:13.116664 9355 heartbeater.cc:347] Connected to a master server at 127.8.213.190:44625
I20240507 08:04:13.117056 9355 heartbeater.cc:464] Registering TS with master...
I20240507 08:04:13.117826 9355 heartbeater.cc:511] Master 127.8.213.190:44625 requested a full tablet report, sending...
I20240507 08:04:13.119827 9078 ts_manager.cc:194] Registered new tserver with Master: 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:13.120492 9046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013311883s
I20240507 08:04:13.121407 9078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55512
I20240507 08:04:13.124516 9046 test_util.cc:268] Using random seed: -1582481524
I20240507 08:04:13.186064 9078 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55524:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:04:13.191496 9078 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:04:13.286314 9171 tablet_service.cc:1467] Processing CreateTablet for tablet 969804d381cd4ba8853bfa7deb87579c (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240507 08:04:13.288070 9171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969804d381cd4ba8853bfa7deb87579c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.291299 9320 tablet_service.cc:1467] Processing CreateTablet for tablet 969804d381cd4ba8853bfa7deb87579c (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240507 08:04:13.292719 9320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969804d381cd4ba8853bfa7deb87579c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.292590 9319 tablet_service.cc:1467] Processing CreateTablet for tablet 9f214d04773142d9aacd663ef367626d (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240507 08:04:13.293534 9170 tablet_service.cc:1467] Processing CreateTablet for tablet 9f214d04773142d9aacd663ef367626d (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240507 08:04:13.294853 9170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f214d04773142d9aacd663ef367626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.294997 9318 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcea0b306f243e8bd5e0be1ae77dfb4 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240507 08:04:13.297336 9317 tablet_service.cc:1467] Processing CreateTablet for tablet 436a0b9810874735b38cb8c2841cb3df (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240507 08:04:13.300778 9319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f214d04773142d9aacd663ef367626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.309955 9164 tablet_service.cc:1467] Processing CreateTablet for tablet e89882d5345e46ce8d753d52521433fe (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240507 08:04:13.312253 9318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcea0b306f243e8bd5e0be1ae77dfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.313285 9165 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3bd791b3ec42c39dbac6038622cd0d (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240507 08:04:13.315455 9164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89882d5345e46ce8d753d52521433fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.320216 9167 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0b5f858a3b4cf1aa3eb448b10955f2 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240507 08:04:13.313184 9168 tablet_service.cc:1467] Processing CreateTablet for tablet 436a0b9810874735b38cb8c2841cb3df (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240507 08:04:13.314769 9169 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcea0b306f243e8bd5e0be1ae77dfb4 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240507 08:04:13.311944 9316 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0b5f858a3b4cf1aa3eb448b10955f2 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240507 08:04:13.321287 9166 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac37475e5734e2eab194a2606deb8a7 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240507 08:04:13.324658 9163 tablet_service.cc:1467] Processing CreateTablet for tablet 239cea47eaaf4d9cbd8b0e25ea7a0e6f (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240507 08:04:13.318591 9314 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3bd791b3ec42c39dbac6038622cd0d (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240507 08:04:13.326072 9165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3bd791b3ec42c39dbac6038622cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.323700 9315 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac37475e5734e2eab194a2606deb8a7 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240507 08:04:13.332080 9245 tablet_service.cc:1467] Processing CreateTablet for tablet 9f214d04773142d9aacd663ef367626d (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20240507 08:04:13.330025 9317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436a0b9810874735b38cb8c2841cb3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.338420 9168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436a0b9810874735b38cb8c2841cb3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.339187 9245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f214d04773142d9aacd663ef367626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.341809 9312 tablet_service.cc:1467] Processing CreateTablet for tablet 239cea47eaaf4d9cbd8b0e25ea7a0e6f (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240507 08:04:13.343410 9163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239cea47eaaf4d9cbd8b0e25ea7a0e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.347931 9242 tablet_service.cc:1467] Processing CreateTablet for tablet 7e0b5f858a3b4cf1aa3eb448b10955f2 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20240507 08:04:13.349367 9242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0b5f858a3b4cf1aa3eb448b10955f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.350590 9241 tablet_service.cc:1467] Processing CreateTablet for tablet 9ac37475e5734e2eab194a2606deb8a7 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20240507 08:04:13.352034 9241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac37475e5734e2eab194a2606deb8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.341747 9313 tablet_service.cc:1467] Processing CreateTablet for tablet e89882d5345e46ce8d753d52521433fe (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240507 08:04:13.343933 9316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0b5f858a3b4cf1aa3eb448b10955f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.340114 9244 tablet_service.cc:1467] Processing CreateTablet for tablet 5dcea0b306f243e8bd5e0be1ae77dfb4 (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20240507 08:04:13.362212 9315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac37475e5734e2eab194a2606deb8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.362581 9246 tablet_service.cc:1467] Processing CreateTablet for tablet 969804d381cd4ba8853bfa7deb87579c (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION VALUES < 238609294
I20240507 08:04:13.364555 9169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcea0b306f243e8bd5e0be1ae77dfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.366315 9240 tablet_service.cc:1467] Processing CreateTablet for tablet 9b3bd791b3ec42c39dbac6038622cd0d (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20240507 08:04:13.369488 9243 tablet_service.cc:1467] Processing CreateTablet for tablet 436a0b9810874735b38cb8c2841cb3df (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20240507 08:04:13.372542 9313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89882d5345e46ce8d753d52521433fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.374387 9246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969804d381cd4ba8853bfa7deb87579c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.370978 9239 tablet_service.cc:1467] Processing CreateTablet for tablet e89882d5345e46ce8d753d52521433fe (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20240507 08:04:13.376152 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dcea0b306f243e8bd5e0be1ae77dfb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.377025 9240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3bd791b3ec42c39dbac6038622cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.379653 9312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239cea47eaaf4d9cbd8b0e25ea7a0e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.378597 9166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ac37475e5734e2eab194a2606deb8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.387192 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 436a0b9810874735b38cb8c2841cb3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.384570 9238 tablet_service.cc:1467] Processing CreateTablet for tablet 239cea47eaaf4d9cbd8b0e25ea7a0e6f (DEFAULT_TABLE table=test-workload [id=9c96df2c95a14033944fa9585b3cd090]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20240507 08:04:13.390419 9239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89882d5345e46ce8d753d52521433fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.393118 9314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b3bd791b3ec42c39dbac6038622cd0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.395622 9238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 239cea47eaaf4d9cbd8b0e25ea7a0e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:13.410039 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e0b5f858a3b4cf1aa3eb448b10955f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:04:13.523413 9126 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:04:13.523698 9373 tablet_bootstrap.cc:492] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:13.618885 9375 tablet_bootstrap.cc:492] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:13.623878 9374 tablet_bootstrap.cc:492] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:13.639396 9373 tablet_bootstrap.cc:654] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.640676 9375 tablet_bootstrap.cc:654] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.659163 9375 tablet_bootstrap.cc:492] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:13.659655 9375 ts_tablet_manager.cc:1367] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.083s user 0.004s sys 0.009s
I20240507 08:04:13.660187 9374 tablet_bootstrap.cc:654] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.660296 9373 tablet_bootstrap.cc:492] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:13.660787 9373 ts_tablet_manager.cc:1367] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.144s user 0.007s sys 0.013s
I20240507 08:04:13.662443 9375 raft_consensus.cc:348] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.663416 9375 raft_consensus.cc:374] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.663393 9373 raft_consensus.cc:348] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.663969 9375 raft_consensus.cc:729] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.664255 9373 raft_consensus.cc:374] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.664600 9373 raft_consensus.cc:729] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.664896 9375 consensus_queue.cc:260] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.665352 9373 consensus_queue.cc:260] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.672577 9374 tablet_bootstrap.cc:492] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:13.673102 9374 ts_tablet_manager.cc:1367] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.005s
I20240507 08:04:13.687227 9375 ts_tablet_manager.cc:1398] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.027s user 0.019s sys 0.008s
I20240507 08:04:13.688006 9355 heartbeater.cc:503] Master 127.8.213.190:44625 was elected leader, sending a full tablet report...
I20240507 08:04:13.680611 9373 ts_tablet_manager.cc:1398] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.020s user 0.009s sys 0.005s
I20240507 08:04:13.673681 9206 heartbeater.cc:503] Master 127.8.213.190:44625 was elected leader, sending a full tablet report...
I20240507 08:04:13.689055 9375 tablet_bootstrap.cc:492] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:13.691512 9374 raft_consensus.cc:348] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.692452 9374 raft_consensus.cc:374] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.692819 9374 raft_consensus.cc:729] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.693667 9374 consensus_queue.cc:260] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.695259 9375 tablet_bootstrap.cc:654] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.704300 9374 ts_tablet_manager.cc:1398] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.031s user 0.020s sys 0.009s
I20240507 08:04:13.705359 9281 heartbeater.cc:503] Master 127.8.213.190:44625 was elected leader, sending a full tablet report...
I20240507 08:04:13.708323 9375 tablet_bootstrap.cc:492] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:13.708802 9375 ts_tablet_manager.cc:1367] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20240507 08:04:13.708815 9373 tablet_bootstrap.cc:492] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:13.711906 9375 raft_consensus.cc:348] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.712728 9375 raft_consensus.cc:374] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.713740 9375 raft_consensus.cc:729] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.714566 9375 consensus_queue.cc:260] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.713244 9374 tablet_bootstrap.cc:492] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:13.722352 9375 ts_tablet_manager.cc:1398] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20240507 08:04:13.723326 9375 tablet_bootstrap.cc:492] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:13.723836 9373 tablet_bootstrap.cc:654] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.725498 9379 raft_consensus.cc:482] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:13.726117 9379 raft_consensus.cc:504] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.725477 9381 raft_consensus.cc:482] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:13.729925 9375 tablet_bootstrap.cc:654] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.730876 9379 leader_election.cc:288] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:13.729799 9381 raft_consensus.cc:504] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.743999 9380 raft_consensus.cc:482] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:13.744519 9380 raft_consensus.cc:504] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.751350 9381 leader_election.cc:288] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:13.764106 9380 leader_election.cc:288] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:13.766523 9375 tablet_bootstrap.cc:492] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:13.767030 9375 ts_tablet_manager.cc:1367] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.044s user 0.023s sys 0.005s
W20240507 08:04:13.772584 9207 tablet.cc:2369] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:04:13.773203 9375 raft_consensus.cc:348] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.773983 9375 raft_consensus.cc:374] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.774277 9375 raft_consensus.cc:729] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.774987 9375 consensus_queue.cc:260] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.777223 9375 ts_tablet_manager.cc:1398] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240507 08:04:13.782825 9375 tablet_bootstrap.cc:492] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:13.776994 9374 tablet_bootstrap.cc:654] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.793495 9373 tablet_bootstrap.cc:492] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:13.794018 9373 ts_tablet_manager.cc:1367] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.007s
I20240507 08:04:13.796881 9375 tablet_bootstrap.cc:654] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.800168 9374 tablet_bootstrap.cc:492] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:13.800874 9374 ts_tablet_manager.cc:1367] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.088s user 0.008s sys 0.006s
I20240507 08:04:13.797142 9373 raft_consensus.cc:348] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.809830 9373 raft_consensus.cc:374] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.810231 9373 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.816949 9373 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.818732 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:13.819553 9181 raft_consensus.cc:2454] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 0.
I20240507 08:04:13.820297 9373 ts_tablet_manager.cc:1398] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20240507 08:04:13.821887 9295 leader_election.cc:302] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:13.822702 9380 raft_consensus.cc:2781] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:13.823040 9380 raft_consensus.cc:482] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:13.823343 9380 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:13.826957 9373 tablet_bootstrap.cc:492] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:13.831095 9374 raft_consensus.cc:348] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.841468 9374 raft_consensus.cc:374] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.841920 9374 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.842837 9374 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.846544 9374 ts_tablet_manager.cc:1398] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.045s user 0.009s sys 0.000s
I20240507 08:04:13.847800 9374 tablet_bootstrap.cc:492] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:13.859086 9379 raft_consensus.cc:482] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:13.859740 9379 raft_consensus.cc:504] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.865425 9379 leader_election.cc:288] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:13.867843 9375 tablet_bootstrap.cc:492] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:13.862845 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:13.868516 9375 ts_tablet_manager.cc:1367] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.086s user 0.023s sys 0.030s
I20240507 08:04:13.870330 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:13.875872 9375 raft_consensus.cc:348] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.876672 9375 raft_consensus.cc:374] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:13.877349 9375 raft_consensus.cc:729] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:13.877758 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:13.878062 9375 consensus_queue.cc:260] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.878573 9256 raft_consensus.cc:2454] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 0.
I20240507 08:04:13.880414 9375 ts_tablet_manager.cc:1398] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20240507 08:04:13.871361 9380 raft_consensus.cc:504] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
W20240507 08:04:13.883728 9293 leader_election.cc:341] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:13.886994 9375 tablet_bootstrap.cc:492] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:13.893451 9380 leader_election.cc:288] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:13.894001 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:13.894934 9181 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:13.896274 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930"
W20240507 08:04:13.896298 9144 leader_election.cc:341] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:13.901695 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:13.903666 9330 raft_consensus.cc:2379] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 865ec2faa51540809fa31393e6a637d6 in current term 1: Already voted for candidate 2edb4bd3d9cf40b193311bf1e44c20da in this term.
I20240507 08:04:13.905166 9373 tablet_bootstrap.cc:654] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:13.905500 9374 tablet_bootstrap.cc:654] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
W20240507 08:04:13.900319 9293 leader_election.cc:341] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:13.906446 9144 leader_election.cc:302] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:13.907881 9146 leader_election.cc:302] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6; no voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930
I20240507 08:04:13.908787 9379 raft_consensus.cc:2781] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:13.909152 9379 raft_consensus.cc:482] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:13.910645 9379 raft_consensus.cc:3037] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:13.911998 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:13.924130 9375 tablet_bootstrap.cc:654] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
W20240507 08:04:13.937294 9146 leader_election.cc:341] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:13.936787 9181 raft_consensus.cc:2454] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 1.
I20240507 08:04:13.937816 9396 raft_consensus.cc:482] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:13.940582 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:13.942353 9295 leader_election.cc:302] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters: e251508eda9d45e7acaf131d69144930
I20240507 08:04:13.944305 9380 raft_consensus.cc:2781] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:13.942673 9396 raft_consensus.cc:504] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.941936 9379 raft_consensus.cc:504] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:13.955552 9380 raft_consensus.cc:686] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:13.957428 9180 raft_consensus.cc:2379] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e251508eda9d45e7acaf131d69144930 in current term 1: Already voted for candidate 865ec2faa51540809fa31393e6a637d6 in this term.
I20240507 08:04:13.957973 9380 consensus_queue.cc:237] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:13.972751 9397 raft_consensus.cc:2726] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:13.977871 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:13.977942 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:13.984417 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:13.986707 9255 raft_consensus.cc:3037] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:13.988082 9373 tablet_bootstrap.cc:492] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
W20240507 08:04:13.987324 9146 leader_election.cc:341] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:13.976606 9396 leader_election.cc:288] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:13.989627 9373 ts_tablet_manager.cc:1367] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.164s user 0.033s sys 0.013s
I20240507 08:04:13.978878 9379 leader_election.cc:288] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 election: Requested vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
W20240507 08:04:13.990630 9144 leader_election.cc:341] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:13.992154 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:13.993885 9144 leader_election.cc:302] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6; no voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930
I20240507 08:04:13.997936 9375 tablet_bootstrap.cc:492] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:13.994269 9374 tablet_bootstrap.cc:492] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.000396 9374 ts_tablet_manager.cc:1367] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.153s user 0.020s sys 0.001s
I20240507 08:04:13.998592 9375 ts_tablet_manager.cc:1367] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.114s user 0.010s sys 0.016s
I20240507 08:04:14.005152 9379 raft_consensus.cc:2726] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:14.005178 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
W20240507 08:04:14.009122 9221 leader_election.cc:341] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.009752 9221 leader_election.cc:302] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e251508eda9d45e7acaf131d69144930; no voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6
W20240507 08:04:14.012130 9146 leader_election.cc:341] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.013245 9255 raft_consensus.cc:2454] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 1.
I20240507 08:04:14.014369 9144 leader_election.cc:302] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters: 2edb4bd3d9cf40b193311bf1e44c20da
I20240507 08:04:14.014950 9374 raft_consensus.cc:348] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.015422 9397 raft_consensus.cc:2781] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.015695 9374 raft_consensus.cc:374] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.015997 9374 raft_consensus.cc:729] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.016687 9397 raft_consensus.cc:686] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:14.016676 9374 consensus_queue.cc:260] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.017982 9381 raft_consensus.cc:2726] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:14.016515 9078 catalog_manager.cc:5484] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 1 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.017766 9397 consensus_queue.cc:237] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.022356 9375 raft_consensus.cc:348] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.023231 9375 raft_consensus.cc:374] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.023670 9375 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.024472 9379 raft_consensus.cc:482] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.025094 9379 raft_consensus.cc:504] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.026036 9375 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.011533 9373 raft_consensus.cc:348] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.027001 9379 leader_election.cc:288] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.029515 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:14.030283 9373 raft_consensus.cc:374] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.030609 9373 raft_consensus.cc:729] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
W20240507 08:04:14.031133 9146 leader_election.cc:341] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.031319 9373 consensus_queue.cc:260] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.032756 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:14.034472 9373 ts_tablet_manager.cc:1398] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.044s user 0.008s sys 0.001s
I20240507 08:04:14.035476 9373 tablet_bootstrap.cc:492] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
W20240507 08:04:14.035351 9144 leader_election.cc:341] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.035818 9144 leader_election.cc:302] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6; no voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.036536 9379 raft_consensus.cc:2726] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:14.040849 9375 ts_tablet_manager.cc:1398] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.039s user 0.003s sys 0.005s
I20240507 08:04:14.043112 9375 tablet_bootstrap.cc:492] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:14.048558 9374 ts_tablet_manager.cc:1398] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.048s user 0.003s sys 0.012s
I20240507 08:04:14.050590 9374 tablet_bootstrap.cc:492] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:14.056824 9375 tablet_bootstrap.cc:654] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.054683 9076 catalog_manager.cc:5484] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 1 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.059643 9373 tablet_bootstrap.cc:654] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.069478 9374 tablet_bootstrap.cc:654] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.085227 9373 tablet_bootstrap.cc:492] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:14.085831 9373 ts_tablet_manager.cc:1367] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.010s
I20240507 08:04:14.087180 9375 tablet_bootstrap.cc:492] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:14.088497 9375 ts_tablet_manager.cc:1367] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.009s
I20240507 08:04:14.088851 9373 raft_consensus.cc:348] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.089754 9373 raft_consensus.cc:374] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.090170 9373 raft_consensus.cc:729] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.091032 9373 consensus_queue.cc:260] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.092686 9375 raft_consensus.cc:348] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.093449 9375 raft_consensus.cc:374] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.093566 9373 ts_tablet_manager.cc:1398] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240507 08:04:14.093786 9375 raft_consensus.cc:729] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.094542 9373 tablet_bootstrap.cc:492] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:14.094508 9375 consensus_queue.cc:260] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.100077 9375 ts_tablet_manager.cc:1398] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240507 08:04:14.100693 9374 tablet_bootstrap.cc:492] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.101047 9375 tablet_bootstrap.cc:492] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:14.101727 9374 ts_tablet_manager.cc:1367] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.010s
I20240507 08:04:14.102059 9373 tablet_bootstrap.cc:654] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.104552 9374 raft_consensus.cc:348] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.105350 9374 raft_consensus.cc:374] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.105681 9374 raft_consensus.cc:729] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.106392 9374 consensus_queue.cc:260] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.108678 9374 ts_tablet_manager.cc:1398] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20240507 08:04:14.109643 9374 tablet_bootstrap.cc:492] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:14.110740 9375 tablet_bootstrap.cc:654] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.116039 9373 tablet_bootstrap.cc:492] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:14.116668 9373 ts_tablet_manager.cc:1367] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.008s
I20240507 08:04:14.118471 9397 raft_consensus.cc:482] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.118939 9397 raft_consensus.cc:504] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.122294 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:14.120908 9373 raft_consensus.cc:348] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.129611 9374 tablet_bootstrap.cc:654] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.129463 9373 raft_consensus.cc:374] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:04:14.129882 9144 leader_election.cc:341] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.130071 9373 raft_consensus.cc:729] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.122475 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:14.132084 9373 consensus_queue.cc:260] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.133615 9330 raft_consensus.cc:2454] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 0.
I20240507 08:04:14.135779 9146 leader_election.cc:302] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters: e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.136763 9379 raft_consensus.cc:2781] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.137378 9379 raft_consensus.cc:482] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.137858 9379 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.121054 9397 leader_election.cc:288] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.149677 9373 ts_tablet_manager.cc:1398] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.032s user 0.008s sys 0.002s
I20240507 08:04:14.151060 9373 tablet_bootstrap.cc:492] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:14.146993 9379 raft_consensus.cc:504] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.157047 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:14.157311 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.158144 9330 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:04:14.158696 9144 leader_election.cc:341] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.164485 9374 tablet_bootstrap.cc:492] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.165210 9374 ts_tablet_manager.cc:1367] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.000s
I20240507 08:04:14.166827 9379 leader_election.cc:288] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 election: Requested vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.168299 9375 tablet_bootstrap.cc:492] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:14.168794 9375 ts_tablet_manager.cc:1367] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.068s user 0.029s sys 0.029s
I20240507 08:04:14.168396 9374 raft_consensus.cc:348] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:14.169412 9374 raft_consensus.cc:374] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.170269 9374 raft_consensus.cc:729] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.171147 9374 consensus_queue.cc:260] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:14.172086 9375 raft_consensus.cc:348] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.172865 9375 raft_consensus.cc:374] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.173159 9375 raft_consensus.cc:729] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.169363 9330 raft_consensus.cc:2454] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 1.
I20240507 08:04:14.173848 9375 consensus_queue.cc:260] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.175251 9146 leader_election.cc:302] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters: e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.176124 9375 ts_tablet_manager.cc:1398] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240507 08:04:14.176265 9397 raft_consensus.cc:2781] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.176749 9397 raft_consensus.cc:686] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:14.176924 9373 tablet_bootstrap.cc:654] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.177062 9375 tablet_bootstrap.cc:492] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:14.177547 9397 consensus_queue.cc:237] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.184002 9375 tablet_bootstrap.cc:654] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.192118 9075 catalog_manager.cc:5484] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 1 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.196235 9373 tablet_bootstrap.cc:492] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:14.194998 9374 ts_tablet_manager.cc:1398] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20240507 08:04:14.196580 9375 tablet_bootstrap.cc:492] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:14.197017 9373 ts_tablet_manager.cc:1367] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.016s
I20240507 08:04:14.197166 9375 ts_tablet_manager.cc:1367] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20240507 08:04:14.199748 9374 tablet_bootstrap.cc:492] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:14.200573 9373 raft_consensus.cc:348] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.201324 9373 raft_consensus.cc:374] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.201732 9373 raft_consensus.cc:729] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.199756 9375 raft_consensus.cc:348] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.202152 9375 raft_consensus.cc:374] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.202404 9375 raft_consensus.cc:729] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.206287 9375 consensus_queue.cc:260] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.207897 9374 tablet_bootstrap.cc:654] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.208252 9375 ts_tablet_manager.cc:1398] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240507 08:04:14.202440 9373 consensus_queue.cc:260] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.212497 9373 ts_tablet_manager.cc:1398] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20240507 08:04:14.209165 9375 tablet_bootstrap.cc:492] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6: Bootstrap starting.
I20240507 08:04:14.217267 9373 tablet_bootstrap.cc:492] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:14.225515 9380 raft_consensus.cc:482] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.226084 9380 raft_consensus.cc:504] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.227869 9375 tablet_bootstrap.cc:654] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.228183 9380 leader_election.cc:288] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.229285 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:14.229861 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.230172 9255 raft_consensus.cc:2379] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2edb4bd3d9cf40b193311bf1e44c20da in current term 1: Already voted for candidate 865ec2faa51540809fa31393e6a637d6 in this term.
I20240507 08:04:14.231714 9295 leader_election.cc:302] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da; no voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.233048 9373 tablet_bootstrap.cc:654] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.232949 9380 raft_consensus.cc:3037] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.240074 9375 tablet_bootstrap.cc:492] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6: No bootstrap required, opened a new log
I20240507 08:04:14.240767 9375 ts_tablet_manager.cc:1367] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.004s
I20240507 08:04:14.243500 9374 tablet_bootstrap.cc:492] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.243357 9375 raft_consensus.cc:348] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.244174 9374 ts_tablet_manager.cc:1367] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.010s
I20240507 08:04:14.244246 9375 raft_consensus.cc:374] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.244804 9375 raft_consensus.cc:729] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.245621 9375 consensus_queue.cc:260] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.248024 9374 raft_consensus.cc:348] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.252920 9373 tablet_bootstrap.cc:492] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:14.253476 9375 ts_tablet_manager.cc:1398] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6: Time spent starting tablet: real 0.012s user 0.004s sys 0.003s
I20240507 08:04:14.254534 9380 raft_consensus.cc:2726] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:14.259624 9373 ts_tablet_manager.cc:1367] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.004s
I20240507 08:04:14.262439 9373 raft_consensus.cc:348] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.263207 9373 raft_consensus.cc:374] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.263506 9373 raft_consensus.cc:729] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.264183 9373 consensus_queue.cc:260] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.253254 9399 raft_consensus.cc:482] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.265883 9399 raft_consensus.cc:504] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.267163 9380 raft_consensus.cc:482] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.268110 9399 leader_election.cc:288] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.267630 9380 raft_consensus.cc:504] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.268975 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.272980 9180 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 0.
I20240507 08:04:14.273293 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.274271 9181 raft_consensus.cc:2454] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 0.
I20240507 08:04:14.276165 9295 leader_election.cc:302] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:14.277205 9399 raft_consensus.cc:2781] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.277336 9295 leader_election.cc:302] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:14.277606 9399 raft_consensus.cc:482] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.277952 9399 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.259090 9374 raft_consensus.cc:374] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.283586 9374 raft_consensus.cc:729] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.284406 9374 consensus_queue.cc:260] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.286834 9374 ts_tablet_manager.cc:1398] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.042s user 0.010s sys 0.001s
I20240507 08:04:14.286787 9399 raft_consensus.cc:504] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.282954 9422 raft_consensus.cc:2781] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.283090 9380 leader_election.cc:288] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.288091 9422 raft_consensus.cc:482] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.288730 9422 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.288916 9399 leader_election.cc:288] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.289711 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:14.290189 9373 ts_tablet_manager.cc:1398] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.030s user 0.003s sys 0.013s
I20240507 08:04:14.290388 9255 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 0.
I20240507 08:04:14.291230 9373 tablet_bootstrap.cc:492] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:14.292548 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.293148 9181 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.297459 9373 tablet_bootstrap.cc:654] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.302150 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:14.304394 9181 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 1.
I20240507 08:04:14.306210 9295 leader_election.cc:302] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:14.288093 9374 tablet_bootstrap.cc:492] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:14.307298 9380 raft_consensus.cc:2781] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.307857 9380 raft_consensus.cc:686] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:14.308519 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:14.308683 9380 consensus_queue.cc:237] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.309165 9255 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.302714 9422 raft_consensus.cc:504] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.315028 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.315881 9422 leader_election.cc:288] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.315968 9181 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:04:14.318531 9293 leader_election.cc:341] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.320549 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:14.316886 9255 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 1.
W20240507 08:04:14.322803 9293 leader_election.cc:341] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:14.327196 9373 tablet_bootstrap.cc:492] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:14.327715 9373 ts_tablet_manager.cc:1367] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.021s
I20240507 08:04:14.328632 9181 raft_consensus.cc:2454] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 1.
I20240507 08:04:14.329846 9295 leader_election.cc:302] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters: e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.330720 9422 raft_consensus.cc:2781] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.330505 9373 raft_consensus.cc:348] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.331214 9373 raft_consensus.cc:374] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.331254 9422 raft_consensus.cc:686] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:14.331617 9373 raft_consensus.cc:729] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.332058 9422 consensus_queue.cc:237] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.332262 9373 consensus_queue.cc:260] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.334306 9373 ts_tablet_manager.cc:1398] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240507 08:04:14.335188 9373 tablet_bootstrap.cc:492] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: Bootstrap starting.
I20240507 08:04:14.336300 9078 catalog_manager.cc:5484] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 1 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:14.340409 9373 tablet_bootstrap.cc:654] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.342841 9374 tablet_bootstrap.cc:654] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.345089 9405 raft_consensus.cc:482] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.345630 9405 raft_consensus.cc:504] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.347800 9405 leader_election.cc:288] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.350018 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:14.350976 9330 raft_consensus.cc:2454] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 0.
I20240507 08:04:14.352437 9221 leader_election.cc:302] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.353755 9405 raft_consensus.cc:2781] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.353447 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.354121 9405 raft_consensus.cc:482] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.354470 9405 raft_consensus.cc:3037] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.354601 9181 raft_consensus.cc:2454] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 0.
I20240507 08:04:14.357599 9078 catalog_manager.cc:5484] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 1 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:14.359999 9374 tablet_bootstrap.cc:492] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.366101 9374 ts_tablet_manager.cc:1367] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.078s user 0.012s sys 0.003s
I20240507 08:04:14.368949 9374 raft_consensus.cc:348] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.369238 9405 raft_consensus.cc:504] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.369874 9374 raft_consensus.cc:374] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.370347 9374 raft_consensus.cc:729] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.371088 9374 consensus_queue.cc:260] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.371522 9405 leader_election.cc:288] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.361562 9373 tablet_bootstrap.cc:492] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: No bootstrap required, opened a new log
I20240507 08:04:14.373893 9373 ts_tablet_manager.cc:1367] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.013s
I20240507 08:04:14.373771 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.374405 9181 raft_consensus.cc:3037] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.374630 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.375232 9330 raft_consensus.cc:3037] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.376427 9373 raft_consensus.cc:348] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.377106 9373 raft_consensus.cc:374] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.377370 9373 raft_consensus.cc:729] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.377992 9373 consensus_queue.cc:260] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.379936 9373 ts_tablet_manager.cc:1398] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:04:14.373383 9374 ts_tablet_manager.cc:1398] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240507 08:04:14.381251 9374 tablet_bootstrap.cc:492] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:14.382339 9330 raft_consensus.cc:2454] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 1.
I20240507 08:04:14.383262 9255 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.387025 9374 tablet_bootstrap.cc:654] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.389634 9221 leader_election.cc:302] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.391000 9380 consensus_queue.cc:1035] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:14.392769 9405 raft_consensus.cc:2781] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.380952 9181 raft_consensus.cc:2454] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 1.
I20240507 08:04:14.394060 9405 raft_consensus.cc:686] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:14.395053 9405 consensus_queue.cc:237] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.414134 9380 consensus_queue.cc:1035] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.410704 9075 catalog_manager.cc:5484] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 0 to 1, leader changed from <none> to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.422466 9374 tablet_bootstrap.cc:492] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.423094 9374 ts_tablet_manager.cc:1367] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.021s
I20240507 08:04:14.425962 9374 raft_consensus.cc:348] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.426930 9374 raft_consensus.cc:374] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.427274 9374 raft_consensus.cc:729] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.428083 9374 consensus_queue.cc:260] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.438150 9374 ts_tablet_manager.cc:1398] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
I20240507 08:04:14.439105 9374 tablet_bootstrap.cc:492] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: Bootstrap starting.
I20240507 08:04:14.443632 9374 tablet_bootstrap.cc:654] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:14.450703 9374 tablet_bootstrap.cc:492] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: No bootstrap required, opened a new log
I20240507 08:04:14.451176 9374 ts_tablet_manager.cc:1367] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240507 08:04:14.453043 9381 raft_consensus.cc:482] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.453469 9374 raft_consensus.cc:348] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.454252 9374 raft_consensus.cc:374] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:14.453478 9381 raft_consensus.cc:504] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.454524 9374 raft_consensus.cc:729] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Initialized, Role: FOLLOWER
I20240507 08:04:14.455148 9374 consensus_queue.cc:260] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.455879 9381 leader_election.cc:288] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.457216 9374 ts_tablet_manager.cc:1398] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240507 08:04:14.457341 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.457952 9181 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 0.
I20240507 08:04:14.458854 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:14.459466 9330 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 0.
I20240507 08:04:14.461266 9221 leader_election.cc:302] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.462220 9381 raft_consensus.cc:2781] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.462654 9381 raft_consensus.cc:482] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.462958 9381 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.468381 9381 raft_consensus.cc:504] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.470261 9381 leader_election.cc:288] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.471035 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.471207 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.471629 9330 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.471622 9181 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.476011 9330 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 1.
I20240507 08:04:14.476967 9221 leader_election.cc:302] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.477650 9381 raft_consensus.cc:2781] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.477926 9181 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 1.
I20240507 08:04:14.478216 9381 raft_consensus.cc:686] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:14.479007 9381 consensus_queue.cc:237] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.486714 9076 catalog_manager.cc:5484] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 0 to 1, leader changed from <none> to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.496451 9381 raft_consensus.cc:482] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.496877 9381 raft_consensus.cc:504] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.498349 9381 leader_election.cc:288] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.499281 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.499416 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:14.499934 9181 raft_consensus.cc:2454] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 0.
I20240507 08:04:14.499981 9330 raft_consensus.cc:2454] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 0.
I20240507 08:04:14.500953 9221 leader_election.cc:302] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.501613 9381 raft_consensus.cc:2781] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.501950 9381 raft_consensus.cc:482] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.502249 9381 raft_consensus.cc:3037] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.508231 9381 raft_consensus.cc:504] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.510217 9381 leader_election.cc:288] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.511061 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.511270 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.511592 9181 raft_consensus.cc:3037] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.511845 9330 raft_consensus.cc:3037] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.519296 9181 raft_consensus.cc:2454] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 1.
I20240507 08:04:14.519096 9397 consensus_queue.cc:1035] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:14.521154 9330 raft_consensus.cc:2454] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 1.
I20240507 08:04:14.521001 9221 leader_election.cc:302] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.522269 9381 raft_consensus.cc:2781] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.523494 9381 raft_consensus.cc:686] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:14.534054 9381 consensus_queue.cc:237] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.537549 9397 consensus_queue.cc:1035] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240507 08:04:14.539412 9126 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:04:14.552119 9076 catalog_manager.cc:5484] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 0 to 1, leader changed from <none> to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.600335 9399 raft_consensus.cc:482] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.600804 9399 raft_consensus.cc:504] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.602584 9399 leader_election.cc:288] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.603478 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.603669 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930" is_pre_election: true
I20240507 08:04:14.604209 9255 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 0.
I20240507 08:04:14.604187 9181 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 0.
I20240507 08:04:14.605317 9293 leader_election.cc:302] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.606052 9399 raft_consensus.cc:2781] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:14.606389 9399 raft_consensus.cc:482] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:14.606653 9399 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.610880 9399 raft_consensus.cc:504] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.612371 9399 leader_election.cc:288] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.613314 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.613426 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:14.613849 9181 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.614073 9255 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.620398 9256 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.620553 9181 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 1.
I20240507 08:04:14.620816 9255 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 1.
I20240507 08:04:14.621711 9295 leader_election.cc:302] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:14.622596 9399 raft_consensus.cc:2781] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:14.623085 9399 raft_consensus.cc:686] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:14.623920 9399 consensus_queue.cc:237] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.629952 9397 consensus_queue.cc:1035] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:14.641989 9379 consensus_queue.cc:1035] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.636714 9076 catalog_manager.cc:5484] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 0 to 1, leader changed from <none> to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 1 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:14.740164 9329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
mode: GRACEFUL
new_leader_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:42306
I20240507 08:04:14.740712 9381 raft_consensus.cc:482] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:14.740851 9329 raft_consensus.cc:595] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Received request to transfer leadership to TS 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:14.741257 9381 raft_consensus.cc:504] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.743255 9381 leader_election.cc:288] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.744237 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865ec2faa51540809fa31393e6a637d6" is_pre_election: true
I20240507 08:04:14.745016 9181 raft_consensus.cc:2379] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e251508eda9d45e7acaf131d69144930 in current term 1: Already voted for candidate 2edb4bd3d9cf40b193311bf1e44c20da in this term.
I20240507 08:04:14.745113 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" is_pre_election: true
I20240507 08:04:14.746356 9221 leader_election.cc:302] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e251508eda9d45e7acaf131d69144930; no voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:14.747123 9381 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:14.750649 9329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
mode: GRACEFUL
new_leader_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:42306
I20240507 08:04:14.751282 9329 raft_consensus.cc:595] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Received request to transfer leadership to TS 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:14.754778 9381 raft_consensus.cc:2726] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:14.762039 9329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:42306
I20240507 08:04:14.762567 9329 raft_consensus.cc:595] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.770076 9329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:42306
I20240507 08:04:14.770596 9329 raft_consensus.cc:595] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:14.780778 9399 consensus_queue.cc:1035] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.785396 9255 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:60652
I20240507 08:04:14.786186 9255 raft_consensus.cc:595] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Received request to transfer leadership to TS 2edb4bd3d9cf40b193311bf1e44c20da
I20240507 08:04:14.790051 9421 raft_consensus.cc:979] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:14.790501 9421 raft_consensus.cc:1067] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:14.795876 9421 consensus_queue.cc:1035] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.796037 9255 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60648
I20240507 08:04:14.796805 9255 raft_consensus.cc:482] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:14.797173 9255 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.805043 9381 consensus_queue.cc:1035] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.813659 9255 raft_consensus.cc:504] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.817682 9181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:14.818347 9181 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.820719 9255 leader_election.cc:288] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.821352 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.821977 9329 raft_consensus.cc:3032] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:14.823143 9329 raft_consensus.cc:729] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:14.823936 9329 consensus_queue.cc:260] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.827973 9405 consensus_queue.cc:1035] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.829043 9329 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.837765 9380 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.837451 9181 raft_consensus.cc:2454] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:14.846257 9221 leader_election.cc:302] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.847404 9405 raft_consensus.cc:2781] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:14.847883 9405 raft_consensus.cc:686] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:14.848692 9405 consensus_queue.cc:237] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.849208 9329 raft_consensus.cc:2454] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:14.875031 9380 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:14.875154 9075 catalog_manager.cc:5484] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 1 to 2, leader changed from 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.885303 9380 raft_consensus.cc:979] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: : Instructing follower 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:14.885777 9399 raft_consensus.cc:1067] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Signalling peer 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:14.887326 9181 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:59678
I20240507 08:04:14.887986 9181 raft_consensus.cc:482] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:14.888398 9181 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.893224 9181 raft_consensus.cc:504] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.895959 9181 leader_election.cc:288] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 2 election: Requested vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:14.895809 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:14.903905 9256 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.896545 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.904687 9330 raft_consensus.cc:3032] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:14.905004 9330 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:14.905691 9330 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.906679 9330 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.910776 9330 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 2.
I20240507 08:04:14.910827 9256 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 2.
I20240507 08:04:14.911834 9146 leader_election.cc:302] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:14.912524 9379 raft_consensus.cc:2781] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:14.912899 9379 raft_consensus.cc:686] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:14.913606 9379 consensus_queue.cc:237] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:14.920356 9076 catalog_manager.cc:5484] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 1 to 2, leader changed from 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131) to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:14.946269 9449 consensus_queue.cc:1035] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.956622 9381 consensus_queue.cc:1035] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:14.964774 9399 raft_consensus.cc:979] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: : Instructing follower 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:14.965238 9399 raft_consensus.cc:1067] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Signalling peer 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:14.966928 9179 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:59678
I20240507 08:04:14.967507 9179 raft_consensus.cc:482] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:14.967846 9179 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.976733 9179 raft_consensus.cc:504] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:14.979676 9330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:14.980396 9179 leader_election.cc:288] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 2 election: Requested vote from peers 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:14.980253 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:14.980305 9330 raft_consensus.cc:3032] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:14.981014 9256 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.981564 9330 raft_consensus.cc:729] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:14.982388 9330 consensus_queue.cc:260] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:14.983620 9330 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:14.988256 9256 raft_consensus.cc:2454] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 2.
I20240507 08:04:14.989624 9144 leader_election.cc:302] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:14.990069 9330 raft_consensus.cc:2454] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 2.
I20240507 08:04:14.990504 9379 raft_consensus.cc:2781] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:14.990998 9379 raft_consensus.cc:686] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:14.991803 9379 consensus_queue.cc:237] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:14.998503 9078 catalog_manager.cc:5484] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 1 to 2, leader changed from 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131) to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:15.006805 9421 consensus_queue.cc:1035] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:15.014205 9422 raft_consensus.cc:979] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:15.014667 9422 raft_consensus.cc:1067] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:15.016834 9255 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60648
I20240507 08:04:15.022064 9255 raft_consensus.cc:482] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:15.022480 9255 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:15.022363 9421 consensus_queue.cc:1035] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:15.028932 9255 raft_consensus.cc:504] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:15.034425 9255 leader_election.cc:288] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:15.035781 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:15.036413 9329 raft_consensus.cc:3032] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:15.036729 9329 raft_consensus.cc:729] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:15.037541 9329 consensus_queue.cc:260] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:15.038686 9329 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:15.040135 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:15.043695 9381 consensus_queue.cc:1035] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:15.052085 9329 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:15.053551 9221 leader_election.cc:302] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters: 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:15.054744 9381 raft_consensus.cc:2781] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:15.055205 9381 raft_consensus.cc:686] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:15.055977 9381 consensus_queue.cc:237] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:15.063261 9405 raft_consensus.cc:979] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: : Instructing follower 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:15.063704 9405 raft_consensus.cc:1067] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Signalling peer 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:15.065635 9329 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:42292
I20240507 08:04:15.065032 9076 catalog_manager.cc:5484] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 1 to 2, leader changed from 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:15.066221 9329 raft_consensus.cc:482] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:15.066610 9329 raft_consensus.cc:3037] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:15.068858 9405 consensus_queue.cc:1035] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:15.078323 9329 raft_consensus.cc:504] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:15.081526 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:15.082419 9179 raft_consensus.cc:3037] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:15.084625 9330 raft_consensus.cc:1226] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Rejecting Update request from peer e251508eda9d45e7acaf131d69144930 for earlier term 1. Current term is 2. Ops: []
I20240507 08:04:15.087131 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "239cea47eaaf4d9cbd8b0e25ea7a0e6f" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:15.087766 9255 raft_consensus.cc:3032] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:15.087971 9329 leader_election.cc:288] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 2 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:15.088094 9255 raft_consensus.cc:729] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:15.088491 9405 consensus_queue.cc:1046] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:15.089740 9255 consensus_queue.cc:260] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:15.090932 9179 raft_consensus.cc:2454] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 2.
I20240507 08:04:15.091076 9255 raft_consensus.cc:3037] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:15.093448 9295 leader_election.cc:302] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:15.094390 9421 raft_consensus.cc:2781] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:15.094861 9421 raft_consensus.cc:686] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:15.095620 9421 consensus_queue.cc:237] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:15.097316 9255 raft_consensus.cc:2454] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 2.
I20240507 08:04:15.104501 9076 catalog_manager.cc:5484] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 1 to 2, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 2 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:15.236943 9179 raft_consensus.cc:1261] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.238361 9405 consensus_queue.cc:1035] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.247584 9329 raft_consensus.cc:1261] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.249358 9405 consensus_queue.cc:1035] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.386878 9255 raft_consensus.cc:1261] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.388204 9379 consensus_queue.cc:1035] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.395187 9330 raft_consensus.cc:1261] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.396596 9397 consensus_queue.cc:1035] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.475015 9179 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:15.479671 9179 raft_consensus.cc:1261] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.481081 9405 consensus_queue.cc:1035] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.491220 9330 raft_consensus.cc:1261] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.492805 9449 consensus_queue.cc:1035] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.498243 9329 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.499596 9397 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.508957 9255 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.511292 9396 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.581532 9179 raft_consensus.cc:1261] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.582923 9421 consensus_queue.cc:1035] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:15.592097 9255 raft_consensus.cc:1261] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:15.594640 9380 consensus_queue.cc:1035] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:17.809592 9046 auto_leader_rebalancer-test.cc:261] The leader distribution is
I20240507 08:04:17.810351 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:17.831830 9179 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
mode: GRACEFUL
new_leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:46990
I20240507 08:04:17.832329 9179 raft_consensus.cc:595] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Received request to transfer leadership to TS 2edb4bd3d9cf40b193311bf1e44c20da
I20240507 08:04:17.844024 9255 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:41160
I20240507 08:04:17.844534 9255 raft_consensus.cc:595] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Received request to transfer leadership to TS 2edb4bd3d9cf40b193311bf1e44c20da
I20240507 08:04:17.845525 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240507 08:04:17.846004 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:17.959976 9463 raft_consensus.cc:979] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: : Instructing follower 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:17.960398 9463 raft_consensus.cc:1067] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Signalling peer 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:17.961700 9330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:42288
I20240507 08:04:17.962153 9330 raft_consensus.cc:482] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:17.962442 9330 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:17.966909 9330 raft_consensus.cc:504] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:17.968688 9330 leader_election.cc:288] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 3 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:17.969501 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:17.969548 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:17.970170 9179 raft_consensus.cc:3032] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:17.970250 9255 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:17.970450 9179 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:17.971132 9179 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:17.972013 9179 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:17.974728 9255 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 3.
I20240507 08:04:17.975713 9293 leader_election.cc:302] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:17.976336 9460 raft_consensus.cc:2781] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:17.976347 9179 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 3.
I20240507 08:04:17.977552 9460 raft_consensus.cc:686] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:17.978348 9460 consensus_queue.cc:237] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:17.983696 9467 raft_consensus.cc:979] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: : Instructing follower 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:17.984138 9381 raft_consensus.cc:1067] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Signalling peer 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:17.985460 9330 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:42292
I20240507 08:04:17.986057 9330 raft_consensus.cc:482] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:17.986479 9330 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:17.986171 9076 catalog_manager.cc:5484] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 2 to 3, leader changed from 865ec2faa51540809fa31393e6a637d6 (127.8.213.129) to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 3 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:17.991987 9330 raft_consensus.cc:504] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:17.994944 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:17.995071 9330 leader_election.cc:288] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 2 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:17.995321 9255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:17.995641 9179 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:17.995934 9255 raft_consensus.cc:3032] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:17.996313 9255 raft_consensus.cc:729] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:17.997066 9255 consensus_queue.cc:260] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:17.998391 9255 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:18.002934 9179 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 2.
I20240507 08:04:18.003844 9295 leader_election.cc:302] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:18.003898 9255 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 2.
I20240507 08:04:18.004550 9460 raft_consensus.cc:2781] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:18.005023 9460 raft_consensus.cc:686] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:18.005882 9460 consensus_queue.cc:237] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:18.013664 9078 catalog_manager.cc:5484] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 1 to 2, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 2 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:18.468027 9255 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:18.469200 9460 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:18.474715 9256 raft_consensus.cc:1261] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:18.476236 9460 consensus_queue.cc:1035] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:18.480624 9179 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:18.483681 9476 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:04:18.491454 9180 raft_consensus.cc:1261] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:18.493419 9465 consensus_queue.cc:1035] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:04:18.846815 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:18.851773 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:18.852210 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:19.853132 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:19.859126 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:19.859546 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:20.860342 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:20.865986 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:20.866504 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:21.867504 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:21.872354 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:21.872711 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:22.873450 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:22.878074 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:22.878455 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:23.879249 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:23.884030 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:23.884429 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:24.885193 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:24.891005 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:24.891405 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:25.892235 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:25.897938 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:25.898516 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:26.899312 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:26.904343 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:26.904731 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:27.926636 9046 auto_leader_rebalancer-test.cc:280] The leader distribution is
I20240507 08:04:27.940479 9179 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:33726
I20240507 08:04:27.941210 9179 raft_consensus.cc:595] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:27.946049 9179 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f214d04773142d9aacd663ef367626d"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:33726
I20240507 08:04:27.946473 9179 raft_consensus.cc:595] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:27.950759 9329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:42306
I20240507 08:04:27.951285 9329 raft_consensus.cc:595] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:27.955469 9179 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:33726
I20240507 08:04:27.955935 9179 raft_consensus.cc:595] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:27.964830 9329 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
mode: GRACEFUL
new_leader_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:42306
I20240507 08:04:27.965451 9329 raft_consensus.cc:595] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Received request to transfer leadership to TS e251508eda9d45e7acaf131d69144930
I20240507 08:04:27.992444 9482 raft_consensus.cc:979] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:27.992846 9482 raft_consensus.cc:1067] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:27.994220 9256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f214d04773142d9aacd663ef367626d"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60642
I20240507 08:04:27.994609 9256 raft_consensus.cc:482] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:27.994917 9256 raft_consensus.cc:3037] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:27.999459 9256 raft_consensus.cc:504] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.000908 9256 leader_election.cc:288] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:28.001766 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:28.002285 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f214d04773142d9aacd663ef367626d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:28.002539 9179 raft_consensus.cc:3032] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:28.002849 9179 raft_consensus.cc:729] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:28.002828 9329 raft_consensus.cc:3037] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:28.003722 9179 consensus_queue.cc:260] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.004983 9179 raft_consensus.cc:3037] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:28.009521 9329 raft_consensus.cc:2454] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:28.010133 9482 raft_consensus.cc:979] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.010576 9482 raft_consensus.cc:1067] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.010783 9221 leader_election.cc:302] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:28.011202 9179 raft_consensus.cc:2454] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:28.011729 9467 raft_consensus.cc:2781] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:28.011914 9256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60642
I20240507 08:04:28.012418 9256 raft_consensus.cc:482] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:28.012791 9256 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:28.012859 9467 raft_consensus.cc:686] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:28.013903 9467 consensus_queue.cc:237] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.016893 9256 raft_consensus.cc:504] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.018755 9256 leader_election.cc:288] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:28.019538 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:28.019795 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:28.020067 9179 raft_consensus.cc:3032] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:28.020298 9329 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:28.020355 9179 raft_consensus.cc:729] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:28.021034 9179 consensus_queue.cc:260] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.022121 9179 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:28.023865 9078 catalog_manager.cc:5484] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 1 to 2, leader changed from 865ec2faa51540809fa31393e6a637d6 (127.8.213.129) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:28.028365 9179 raft_consensus.cc:2454] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:28.028362 9329 raft_consensus.cc:2454] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 2.
I20240507 08:04:28.029666 9221 leader_election.cc:302] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:28.030274 9467 raft_consensus.cc:2781] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:28.030839 9467 raft_consensus.cc:686] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:28.031662 9467 consensus_queue.cc:237] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.042516 9078 catalog_manager.cc:5484] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 1 to 2, leader changed from 865ec2faa51540809fa31393e6a637d6 (127.8.213.129) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:28.220136 9482 raft_consensus.cc:979] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.220733 9489 raft_consensus.cc:1067] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.222149 9256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60642
I20240507 08:04:28.222554 9256 raft_consensus.cc:482] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:28.222803 9256 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:28.226841 9256 raft_consensus.cc:504] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:28.228053 9256 leader_election.cc:288] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 3 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:28.228788 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:28.229015 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969804d381cd4ba8853bfa7deb87579c" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:28.229334 9179 raft_consensus.cc:3032] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:28.229617 9329 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:28.229802 9179 raft_consensus.cc:729] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:28.230544 9179 consensus_queue.cc:260] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:28.231649 9179 raft_consensus.cc:3037] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:28.235764 9329 raft_consensus.cc:2454] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 3.
I20240507 08:04:28.236711 9221 leader_election.cc:302] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:28.237205 9179 raft_consensus.cc:2454] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 3.
I20240507 08:04:28.237313 9467 raft_consensus.cc:2781] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:28.237836 9467 raft_consensus.cc:686] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 3 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:28.238674 9467 consensus_queue.cc:237] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } }
I20240507 08:04:28.245092 9078 catalog_manager.cc:5484] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 2 to 3, leader changed from 865ec2faa51540809fa31393e6a637d6 (127.8.213.129) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:28.398993 9460 raft_consensus.cc:979] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.399369 9460 raft_consensus.cc:1067] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.400581 9256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60648
I20240507 08:04:28.401003 9256 raft_consensus.cc:482] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:28.401226 9256 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:28.405252 9256 raft_consensus.cc:504] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.406553 9256 leader_election.cc:288] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 4 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:28.407447 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:28.407509 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:28.407891 9179 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:28.408007 9329 raft_consensus.cc:3032] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:04:28.408326 9329 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:28.408844 9329 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.409802 9329 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:28.411957 9179 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 4.
I20240507 08:04:28.412765 9221 leader_election.cc:302] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:28.413337 9467 raft_consensus.cc:2781] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:04:28.413676 9467 raft_consensus.cc:686] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:28.413821 9329 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 4.
I20240507 08:04:28.414310 9467 consensus_queue.cc:237] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.420709 9078 catalog_manager.cc:5484] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 3 to 4, leader changed from 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 4 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:28.456526 9329 raft_consensus.cc:1261] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:28.457546 9467 consensus_queue.cc:1035] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:28.467134 9179 raft_consensus.cc:1261] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:28.469264 9467 consensus_queue.cc:1035] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:04:28.483076 9179 raft_consensus.cc:1261] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:28.484381 9502 consensus_queue.cc:1035] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:28.494107 9329 raft_consensus.cc:1261] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:28.496078 9502 consensus_queue.cc:1035] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:28.507972 9498 raft_consensus.cc:979] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: : Instructing follower e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.508395 9498 raft_consensus.cc:1067] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Signalling peer e251508eda9d45e7acaf131d69144930 to start an election
I20240507 08:04:28.509827 9256 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
from {username='slave'} at 127.0.0.1:60648
I20240507 08:04:28.510272 9256 raft_consensus.cc:482] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:28.510497 9256 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:28.514349 9256 raft_consensus.cc:504] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.515612 9256 leader_election.cc:288] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 3 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:28.516532 9179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:28.516532 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b3bd791b3ec42c39dbac6038622cd0d" candidate_uuid: "e251508eda9d45e7acaf131d69144930" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:28.517241 9179 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:28.517320 9329 raft_consensus.cc:3032] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:28.517652 9329 raft_consensus.cc:729] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:28.518348 9329 consensus_queue.cc:260] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.519137 9329 raft_consensus.cc:3037] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:28.521523 9179 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 3.
I20240507 08:04:28.522578 9221 leader_election.cc:302] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 865ec2faa51540809fa31393e6a637d6, e251508eda9d45e7acaf131d69144930; no voters:
I20240507 08:04:28.523159 9329 raft_consensus.cc:2454] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e251508eda9d45e7acaf131d69144930 in term 3.
I20240507 08:04:28.523278 9502 raft_consensus.cc:2781] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:28.523775 9502 raft_consensus.cc:686] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 3 LEADER]: Becoming Leader. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:28.524689 9502 consensus_queue.cc:237] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:28.532269 9076 catalog_manager.cc:5484] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 reported cstate change: term changed from 2 to 3, leader changed from 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131) to e251508eda9d45e7acaf131d69144930 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "e251508eda9d45e7acaf131d69144930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:28.818879 9329 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 4 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:04:28.820129 9502 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:04:28.828680 9179 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 4 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:04:28.832304 9502 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:04:28.845387 9179 raft_consensus.cc:1261] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:28.846451 9502 consensus_queue.cc:1035] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:28.854975 9329 raft_consensus.cc:1261] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:28.858551 9502 consensus_queue.cc:1035] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:28.997735 9329 raft_consensus.cc:1261] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:28.998849 9490 consensus_queue.cc:1035] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:29.008872 9179 raft_consensus.cc:1261] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Refusing update from remote peer e251508eda9d45e7acaf131d69144930: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:29.010293 9490 consensus_queue.cc:1035] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:30.991139 9046 auto_leader_rebalancer-test.cc:295] The leader distribution is
I20240507 08:04:30.991775 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:30.998476 9256 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:41160
I20240507 08:04:30.999040 9256 raft_consensus.cc:595] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Received request to transfer leadership to TS 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:31.001348 9256 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:41160
I20240507 08:04:31.001821 9256 raft_consensus.cc:595] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 LEADER]: Received request to transfer leadership to TS 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:31.003696 9256 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:41160
I20240507 08:04:31.004098 9256 raft_consensus.cc:595] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Received request to transfer leadership to TS 865ec2faa51540809fa31393e6a637d6
I20240507 08:04:31.005993 9256 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:41160
I20240507 08:04:31.006395 9256 raft_consensus.cc:595] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Received request to transfer leadership to TS 2edb4bd3d9cf40b193311bf1e44c20da
I20240507 08:04:31.008239 9256 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe"
dest_uuid: "e251508eda9d45e7acaf131d69144930"
mode: GRACEFUL
new_leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:41160
I20240507 08:04:31.008627 9256 raft_consensus.cc:595] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Received request to transfer leadership to TS 2edb4bd3d9cf40b193311bf1e44c20da
I20240507 08:04:31.009503 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:04:31.009925 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:31.074213 9516 raft_consensus.cc:979] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: : Instructing follower 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:31.074666 9490 raft_consensus.cc:1067] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Signalling peer 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:31.075956 9180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:59692
I20240507 08:04:31.076427 9180 raft_consensus.cc:482] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:31.076679 9180 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.081086 9180 raft_consensus.cc:504] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.083158 9180 leader_election.cc:288] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 3 election: Requested vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:31.083921 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:31.084195 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "436a0b9810874735b38cb8c2841cb3df" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:31.084383 9256 raft_consensus.cc:3032] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:31.084697 9329 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.084712 9256 raft_consensus.cc:729] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:31.085352 9256 consensus_queue.cc:260] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.086201 9256 raft_consensus.cc:3037] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.088732 9329 raft_consensus.cc:2454] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 3.
I20240507 08:04:31.089844 9146 leader_election.cc:302] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:31.090106 9256 raft_consensus.cc:2454] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 3.
I20240507 08:04:31.091346 9517 raft_consensus.cc:2781] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:31.092103 9517 raft_consensus.cc:686] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 3 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:31.092765 9517 consensus_queue.cc:237] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.098820 9076 catalog_manager.cc:5484] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 2 to 3, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 3 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:31.108274 9516 raft_consensus.cc:979] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: : Instructing follower 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:31.108702 9490 raft_consensus.cc:1067] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Signalling peer 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:31.110029 9329 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:42292
I20240507 08:04:31.110543 9329 raft_consensus.cc:482] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:31.110872 9329 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.114794 9329 raft_consensus.cc:504] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.116434 9329 leader_election.cc:288] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 3 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:31.117311 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:31.117339 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89882d5345e46ce8d753d52521433fe" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:31.118039 9180 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.118091 9256 raft_consensus.cc:3032] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:31.118489 9256 raft_consensus.cc:729] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:31.119091 9256 consensus_queue.cc:260] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.119938 9256 raft_consensus.cc:3037] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.122624 9180 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 3.
I20240507 08:04:31.123734 9295 leader_election.cc:302] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:31.123905 9256 raft_consensus.cc:2454] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 3.
I20240507 08:04:31.124580 9498 raft_consensus.cc:2781] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:31.125762 9498 raft_consensus.cc:686] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:31.126466 9498 consensus_queue.cc:237] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.132849 9076 catalog_manager.cc:5484] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 2 to 3, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 3 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:31.204720 9516 raft_consensus.cc:979] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: : Instructing follower 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:31.205402 9490 raft_consensus.cc:1067] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Signalling peer 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:31.206976 9180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:59692
I20240507 08:04:31.207579 9180 raft_consensus.cc:482] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:31.207901 9180 raft_consensus.cc:3037] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:31.213754 9180 raft_consensus.cc:504] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.215667 9180 leader_election.cc:288] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 2 election: Requested vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:31.217478 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:31.217765 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e0b5f858a3b4cf1aa3eb448b10955f2" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:31.218358 9256 raft_consensus.cc:3032] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:31.218537 9329 raft_consensus.cc:3037] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:31.218647 9256 raft_consensus.cc:729] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:31.219664 9256 consensus_queue.cc:260] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.220805 9256 raft_consensus.cc:3037] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:31.223906 9329 raft_consensus.cc:2454] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 2.
I20240507 08:04:31.225129 9146 leader_election.cc:302] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:31.226217 9517 raft_consensus.cc:2781] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:31.226576 9517 raft_consensus.cc:686] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:31.227106 9256 raft_consensus.cc:2454] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 2.
I20240507 08:04:31.227524 9517 consensus_queue.cc:237] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.236570 9076 catalog_manager.cc:5484] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 1 to 2, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:31.507483 9516 raft_consensus.cc:979] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: : Instructing follower 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:31.507968 9490 raft_consensus.cc:1067] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Signalling peer 2edb4bd3d9cf40b193311bf1e44c20da to start an election
I20240507 08:04:31.509279 9329 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7"
dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
from {username='slave'} at 127.0.0.1:42292
I20240507 08:04:31.509742 9329 raft_consensus.cc:482] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:31.509965 9329 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.514021 9329 raft_consensus.cc:504] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.515357 9329 leader_election.cc:288] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 3 election: Requested vote from peers 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421), e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579)
I20240507 08:04:31.516142 9180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "865ec2faa51540809fa31393e6a637d6"
I20240507 08:04:31.516454 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ac37475e5734e2eab194a2606deb8a7" candidate_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:31.516681 9180 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.516988 9256 raft_consensus.cc:3032] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:31.517210 9256 raft_consensus.cc:729] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:31.517932 9256 consensus_queue.cc:260] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.518746 9256 raft_consensus.cc:3037] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:31.521710 9180 raft_consensus.cc:2454] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 3.
I20240507 08:04:31.522694 9295 leader_election.cc:302] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:31.522882 9256 raft_consensus.cc:2454] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2edb4bd3d9cf40b193311bf1e44c20da in term 3.
I20240507 08:04:31.523451 9498 raft_consensus.cc:2781] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:31.523960 9498 raft_consensus.cc:686] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Becoming Leader. State: Replica: 2edb4bd3d9cf40b193311bf1e44c20da, State: Running, Role: LEADER
I20240507 08:04:31.524966 9498 consensus_queue.cc:237] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.531314 9076 catalog_manager.cc:5484] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da reported cstate change: term changed from 2 to 3, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131). New cstate: current_term: 3 leader_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:31.562812 9516 raft_consensus.cc:979] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: : Instructing follower 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:31.563235 9490 raft_consensus.cc:1067] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 LEADER]: Signalling peer 865ec2faa51540809fa31393e6a637d6 to start an election
I20240507 08:04:31.564397 9180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4"
dest_uuid: "865ec2faa51540809fa31393e6a637d6"
from {username='slave'} at 127.0.0.1:59692
I20240507 08:04:31.564815 9180 raft_consensus.cc:482] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:31.565062 9180 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:04:31.569222 9180 raft_consensus.cc:504] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.570636 9180 leader_election.cc:288] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 5 election: Requested vote from peers e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579), 2edb4bd3d9cf40b193311bf1e44c20da (127.8.213.131:37689)
I20240507 08:04:31.571483 9256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "e251508eda9d45e7acaf131d69144930"
I20240507 08:04:31.571532 9329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dcea0b306f243e8bd5e0be1ae77dfb4" candidate_uuid: "865ec2faa51540809fa31393e6a637d6" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2edb4bd3d9cf40b193311bf1e44c20da"
I20240507 08:04:31.572024 9256 raft_consensus.cc:3032] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:04:31.572069 9329 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:04:31.572371 9256 raft_consensus.cc:729] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e251508eda9d45e7acaf131d69144930, State: Running, Role: LEADER
I20240507 08:04:31.572963 9256 consensus_queue.cc:260] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.573823 9256 raft_consensus.cc:3037] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:04:31.576294 9329 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 5.
I20240507 08:04:31.577086 9146 leader_election.cc:302] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2edb4bd3d9cf40b193311bf1e44c20da, 865ec2faa51540809fa31393e6a637d6; no voters:
I20240507 08:04:31.577705 9256 raft_consensus.cc:2454] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 865ec2faa51540809fa31393e6a637d6 in term 5.
I20240507 08:04:31.577735 9517 raft_consensus.cc:2781] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:04:31.578238 9517 raft_consensus.cc:686] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 5 LEADER]: Becoming Leader. State: Replica: 865ec2faa51540809fa31393e6a637d6, State: Running, Role: LEADER
I20240507 08:04:31.579007 9517 consensus_queue.cc:237] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } }
I20240507 08:04:31.584847 9076 catalog_manager.cc:5484] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 reported cstate change: term changed from 4 to 5, leader changed from e251508eda9d45e7acaf131d69144930 (127.8.213.130) to 865ec2faa51540809fa31393e6a637d6 (127.8.213.129). New cstate: current_term: 5 leader_uuid: "865ec2faa51540809fa31393e6a637d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:31.638859 9256 raft_consensus.cc:1261] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:31.639962 9517 consensus_queue.cc:1035] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:31.651212 9329 raft_consensus.cc:1261] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:31.654328 9518 consensus_queue.cc:1035] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:04:31.655717 9329 raft_consensus.cc:1261] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:31.656944 9518 consensus_queue.cc:1035] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:31.668618 9256 raft_consensus.cc:1261] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:31.669871 9517 consensus_queue.cc:1035] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:04:31.671959 9180 raft_consensus.cc:1261] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:31.673734 9498 consensus_queue.cc:1035] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:31.676882 9255 raft_consensus.cc:1261] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:31.678273 9498 consensus_queue.cc:1035] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:31.933768 9256 raft_consensus.cc:1261] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:31.934909 9498 consensus_queue.cc:1035] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:31.937088 9180 raft_consensus.cc:1261] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Refusing update from remote peer 2edb4bd3d9cf40b193311bf1e44c20da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:31.938409 9498 consensus_queue.cc:1035] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [LEADER]: Connected to new peer: Peer: permanent_uuid: "865ec2faa51540809fa31393e6a637d6" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 46421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:32.010911 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:32.015758 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:32.016162 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:32.032611 9330 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 5 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:04:32.034010 9532 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2edb4bd3d9cf40b193311bf1e44c20da" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:04:32.041822 9256 raft_consensus.cc:1261] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 5 FOLLOWER]: Refusing update from remote peer 865ec2faa51540809fa31393e6a637d6: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:04:32.043586 9537 consensus_queue.cc:1035] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e251508eda9d45e7acaf131d69144930" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:04:33.017007 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:33.022569 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:33.023047 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:34.023851 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:34.029139 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:34.029559 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:35.030387 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:35.035490 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:35.035921 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:36.036662 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:36.042191 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:36.042557 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:37.043362 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:37.048211 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:37.048632 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:38.049456 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:38.055048 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:38.055490 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:39.056354 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:39.061302 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:39.061776 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:40.062592 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:40.067399 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:40.067811 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:41.091074 9046 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20240507 08:04:41.091747 9046 tablet_server.cc:178] TabletServer@127.8.213.129:0 shutting down...
I20240507 08:04:41.119482 9046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:04:41.120115 9046 tablet_replica.cc:330] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.120883 9046 raft_consensus.cc:2229] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.121397 9046 raft_consensus.cc:2258] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.125276 9046 tablet_replica.cc:330] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.125952 9046 raft_consensus.cc:2229] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.126355 9046 raft_consensus.cc:2258] T 9b3bd791b3ec42c39dbac6038622cd0d P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.128021 9046 tablet_replica.cc:330] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.128597 9046 raft_consensus.cc:2229] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.129433 9046 raft_consensus.cc:2258] T 436a0b9810874735b38cb8c2841cb3df P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240507 08:04:41.129422 9221 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240507 08:04:41.131635 9046 tablet_replica.cc:330] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.132201 9046 raft_consensus.cc:2229] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus shutting down.
W20240507 08:04:41.132550 9221 consensus_peers.cc:485] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 -> Peer 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421): Couldn't send request to peer 865ec2faa51540809fa31393e6a637d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:04:41.132629 9046 raft_consensus.cc:2258] T 9ac37475e5734e2eab194a2606deb8a7 P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.134363 9046 tablet_replica.cc:330] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.134809 9046 raft_consensus.cc:2229] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.135151 9046 raft_consensus.cc:2258] T e89882d5345e46ce8d753d52521433fe P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.136565 9046 tablet_replica.cc:330] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.136996 9046 raft_consensus.cc:2229] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.137336 9046 raft_consensus.cc:2258] T 9f214d04773142d9aacd663ef367626d P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.138857 9046 tablet_replica.cc:330] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.139317 9046 raft_consensus.cc:2229] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 5 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.139974 9046 raft_consensus.cc:2258] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 865ec2faa51540809fa31393e6a637d6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.141433 9046 tablet_replica.cc:330] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.141893 9046 raft_consensus.cc:2229] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.142511 9046 raft_consensus.cc:2258] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 865ec2faa51540809fa31393e6a637d6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.143952 9046 tablet_replica.cc:330] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6: stopping tablet replica
I20240507 08:04:41.144368 9046 raft_consensus.cc:2229] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.144718 9046 raft_consensus.cc:2258] T 969804d381cd4ba8853bfa7deb87579c P 865ec2faa51540809fa31393e6a637d6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20240507 08:04:41.235522 9295 consensus_peers.cc:485] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da -> Peer 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421): Couldn't send request to peer 865ec2faa51540809fa31393e6a637d6. Status: Network error: Client connection negotiation failed: client connection to 127.8.213.129:46421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:04:41.244339 9046 tablet_server.cc:194] TabletServer@127.8.213.129:0 shutdown complete.
I20240507 08:04:41.265132 9046 tablet_server.cc:178] TabletServer@127.8.213.130:0 shutting down...
I20240507 08:04:41.292708 9046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:04:41.293546 9046 tablet_replica.cc:330] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.294382 9046 raft_consensus.cc:2229] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.295053 9046 raft_consensus.cc:2258] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.296905 9046 tablet_replica.cc:330] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.297747 9046 raft_consensus.cc:2229] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.298281 9046 raft_consensus.cc:2258] T 9ac37475e5734e2eab194a2606deb8a7 P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.299995 9046 tablet_replica.cc:330] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.300469 9046 raft_consensus.cc:2229] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.300938 9046 raft_consensus.cc:2258] T e89882d5345e46ce8d753d52521433fe P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.303553 9046 tablet_replica.cc:330] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.304018 9046 raft_consensus.cc:2229] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.304551 9046 raft_consensus.cc:2258] T 436a0b9810874735b38cb8c2841cb3df P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.306444 9046 tablet_replica.cc:330] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930: stopping tablet replica
W20240507 08:04:41.306952 9295 consensus_peers.cc:485] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da -> Peer 865ec2faa51540809fa31393e6a637d6 (127.8.213.129:46421): Couldn't send request to peer 865ec2faa51540809fa31393e6a637d6. Status: Network error: Client connection negotiation failed: client connection to 127.8.213.129:46421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:04:41.307159 9046 raft_consensus.cc:2229] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.308822 9046 raft_consensus.cc:2258] T 9b3bd791b3ec42c39dbac6038622cd0d P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.311009 9046 tablet_replica.cc:330] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.311479 9046 raft_consensus.cc:2229] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.312508 9046 raft_consensus.cc:2258] T 969804d381cd4ba8853bfa7deb87579c P e251508eda9d45e7acaf131d69144930 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.314139 9046 tablet_replica.cc:330] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.314615 9046 raft_consensus.cc:2229] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.315039 9046 raft_consensus.cc:2258] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.316660 9046 tablet_replica.cc:330] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.317111 9046 raft_consensus.cc:2229] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.317565 9046 raft_consensus.cc:2258] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P e251508eda9d45e7acaf131d69144930 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.319694 9046 tablet_replica.cc:330] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930: stopping tablet replica
I20240507 08:04:41.320138 9046 raft_consensus.cc:2229] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.320869 9046 raft_consensus.cc:2258] T 9f214d04773142d9aacd663ef367626d P e251508eda9d45e7acaf131d69144930 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.349715 9046 tablet_server.cc:194] TabletServer@127.8.213.130:0 shutdown complete.
I20240507 08:04:41.366073 9046 tablet_server.cc:178] TabletServer@127.8.213.131:0 shutting down...
W20240507 08:04:41.385301 9293 consensus_peers.cc:485] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da -> Peer e251508eda9d45e7acaf131d69144930 (127.8.213.130:42579): Couldn't send request to peer e251508eda9d45e7acaf131d69144930. Status: Network error: Client connection negotiation failed: client connection to 127.8.213.130:42579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:04:41.391212 9046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:04:41.391985 9046 tablet_replica.cc:330] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.392604 9046 raft_consensus.cc:2229] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.393383 9046 raft_consensus.cc:2258] T 239cea47eaaf4d9cbd8b0e25ea7a0e6f P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.395145 9046 tablet_replica.cc:330] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.395656 9046 raft_consensus.cc:2229] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.396620 9046 raft_consensus.cc:2258] T e89882d5345e46ce8d753d52521433fe P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.398213 9046 tablet_replica.cc:330] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.398694 9046 raft_consensus.cc:2229] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.399492 9046 raft_consensus.cc:2258] T 9ac37475e5734e2eab194a2606deb8a7 P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.401059 9046 tablet_replica.cc:330] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.401546 9046 raft_consensus.cc:2229] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.402001 9046 raft_consensus.cc:2258] T 9b3bd791b3ec42c39dbac6038622cd0d P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.403672 9046 tablet_replica.cc:330] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.404105 9046 raft_consensus.cc:2229] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.404489 9046 raft_consensus.cc:2258] T 436a0b9810874735b38cb8c2841cb3df P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.405975 9046 tablet_replica.cc:330] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.406419 9046 raft_consensus.cc:2229] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.406811 9046 raft_consensus.cc:2258] T 9f214d04773142d9aacd663ef367626d P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.408300 9046 tablet_replica.cc:330] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.408730 9046 raft_consensus.cc:2229] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.409143 9046 raft_consensus.cc:2258] T 7e0b5f858a3b4cf1aa3eb448b10955f2 P 2edb4bd3d9cf40b193311bf1e44c20da [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.410573 9046 tablet_replica.cc:330] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.410982 9046 raft_consensus.cc:2229] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 5 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.411398 9046 raft_consensus.cc:2258] T 5dcea0b306f243e8bd5e0be1ae77dfb4 P 2edb4bd3d9cf40b193311bf1e44c20da [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.412868 9046 tablet_replica.cc:330] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da: stopping tablet replica
I20240507 08:04:41.413273 9046 raft_consensus.cc:2229] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:04:41.413743 9046 raft_consensus.cc:2258] T 969804d381cd4ba8853bfa7deb87579c P 2edb4bd3d9cf40b193311bf1e44c20da [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.440598 9046 tablet_server.cc:194] TabletServer@127.8.213.131:0 shutdown complete.
I20240507 08:04:41.456329 9046 master.cc:537] Master@127.8.213.190:44625 shutting down...
I20240507 08:04:41.600215 9046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:04:41.600808 9046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:04:41.601111 9046 tablet_replica.cc:330] T 00000000000000000000000000000000 P b2b3a38c57604f748496b35b36ae24e1: stopping tablet replica
I20240507 08:04:41.618744 9046 master.cc:552] Master@127.8.213.190:44625 shutdown complete.
2edb4bd3d9cf40b193311bf1e44c20da 1
865ec2faa51540809fa31393e6a637d6 4
e251508eda9d45e7acaf131d69144930 4
2edb4bd3d9cf40b193311bf1e44c20da 3
865ec2faa51540809fa31393e6a637d6 3
e251508eda9d45e7acaf131d69144930 3
2edb4bd3d9cf40b193311bf1e44c20da 1
865ec2faa51540809fa31393e6a637d6 0
e251508eda9d45e7acaf131d69144930 8
2edb4bd3d9cf40b193311bf1e44c20da 3
865ec2faa51540809fa31393e6a637d6 3
e251508eda9d45e7acaf131d69144930 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30541 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20240507 08:04:41.659341 9046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.213.190:44427
I20240507 08:04:41.660449 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:41.666435 9558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:41.666980 9559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:41.667569 9561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:41.668161 9046 server_base.cc:1034] running on GCE node
I20240507 08:04:41.669073 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:41.669265 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:41.669397 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069081669386 us; error 0 us; skew 500 ppm
I20240507 08:04:41.669982 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:41.672379 9046 webserver.cc:460] Webserver started at http://127.8.213.190:36373/ using document root <none> and password file <none>
I20240507 08:04:41.672829 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:41.672995 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:41.673269 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:41.674464 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/master-0-root/instance:
uuid: "e86bc57feda242dda37183adf4ad14f7"
format_stamp: "Formatted at 2024-05-07 08:04:41 on dist-test-slave-ls56"
I20240507 08:04:41.678923 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240507 08:04:41.682116 9566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:41.682802 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240507 08:04:41.683063 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/master-0-root
uuid: "e86bc57feda242dda37183adf4ad14f7"
format_stamp: "Formatted at 2024-05-07 08:04:41 on dist-test-slave-ls56"
I20240507 08:04:41.683324 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:41.693392 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:41.694617 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:41.730872 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.190:44427
I20240507 08:04:41.730963 9617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.190:44427 every 8 connection(s)
I20240507 08:04:41.734669 9618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:41.743714 9618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7: Bootstrap starting.
I20240507 08:04:41.747864 9618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:41.751734 9618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7: No bootstrap required, opened a new log
I20240507 08:04:41.753702 9618 raft_consensus.cc:348] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86bc57feda242dda37183adf4ad14f7" member_type: VOTER }
I20240507 08:04:41.754269 9618 raft_consensus.cc:374] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:41.754508 9618 raft_consensus.cc:729] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86bc57feda242dda37183adf4ad14f7, State: Initialized, Role: FOLLOWER
I20240507 08:04:41.755201 9618 consensus_queue.cc:260] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86bc57feda242dda37183adf4ad14f7" member_type: VOTER }
I20240507 08:04:41.755690 9618 raft_consensus.cc:388] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:04:41.755909 9618 raft_consensus.cc:482] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:04:41.756191 9618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:41.759941 9618 raft_consensus.cc:504] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86bc57feda242dda37183adf4ad14f7" member_type: VOTER }
I20240507 08:04:41.760407 9618 leader_election.cc:302] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e86bc57feda242dda37183adf4ad14f7; no voters:
I20240507 08:04:41.761487 9618 leader_election.cc:288] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 08:04:41.761893 9621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:41.763253 9621 raft_consensus.cc:686] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 1 LEADER]: Becoming Leader. State: Replica: e86bc57feda242dda37183adf4ad14f7, State: Running, Role: LEADER
I20240507 08:04:41.763926 9621 consensus_queue.cc:237] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86bc57feda242dda37183adf4ad14f7" member_type: VOTER }
I20240507 08:04:41.764475 9618 sys_catalog.cc:564] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:04:41.770890 9623 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e86bc57feda242dda37183adf4ad14f7. Latest consensus state: current_term: 1 leader_uuid: "e86bc57feda242dda37183adf4ad14f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86bc57feda242dda37183adf4ad14f7" member_type: VOTER } }
I20240507 08:04:41.771543 9623 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [sys.catalog]: This master's current role is: LEADER
I20240507 08:04:41.770766 9622 sys_catalog.cc:455] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e86bc57feda242dda37183adf4ad14f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e86bc57feda242dda37183adf4ad14f7" member_type: VOTER } }
I20240507 08:04:41.772786 9622 sys_catalog.cc:458] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [sys.catalog]: This master's current role is: LEADER
I20240507 08:04:41.773456 9630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:04:41.780884 9630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:04:41.781849 9046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:04:41.789094 9630 catalog_manager.cc:1347] Generated new cluster ID: f97c3a957a6b4a4dbebce785aa823bea
I20240507 08:04:41.789391 9630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:04:41.805299 9630 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:04:41.806499 9630 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:04:41.816613 9630 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7: Generated new TSK 0
I20240507 08:04:41.817198 9630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:04:41.848522 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:41.855406 9639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:41.857270 9640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:41.859431 9642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:41.860306 9046 server_base.cc:1034] running on GCE node
I20240507 08:04:41.861055 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:41.861238 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:41.861397 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069081861378 us; error 0 us; skew 500 ppm
I20240507 08:04:41.861963 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:41.866505 9046 webserver.cc:460] Webserver started at http://127.8.213.129:33359/ using document root <none> and password file <none>
I20240507 08:04:41.867349 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:41.867518 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:41.867767 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:41.868805 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-0-root/instance:
uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
format_stamp: "Formatted at 2024-05-07 08:04:41 on dist-test-slave-ls56"
I20240507 08:04:41.873025 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240507 08:04:41.876195 9647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:41.876873 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240507 08:04:41.877128 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-0-root
uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
format_stamp: "Formatted at 2024-05-07 08:04:41 on dist-test-slave-ls56"
I20240507 08:04:41.877377 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:41.904306 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:41.905462 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:41.906960 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:04:41.909123 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:04:41.909292 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:41.909536 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:04:41.909715 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240507 08:04:41.949293 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.129:40337
I20240507 08:04:41.949383 9709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.129:40337 every 8 connection(s)
I20240507 08:04:41.953718 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:41.962198 9715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:41.961395 9714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:41.963613 9717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:41.964802 9046 server_base.cc:1034] running on GCE node
I20240507 08:04:41.966351 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:41.966621 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:41.966830 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069081966810 us; error 0 us; skew 500 ppm
I20240507 08:04:41.967355 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:41.967648 9710 heartbeater.cc:347] Connected to a master server at 127.8.213.190:44427
I20240507 08:04:41.968030 9710 heartbeater.cc:464] Registering TS with master...
I20240507 08:04:41.968887 9710 heartbeater.cc:511] Master 127.8.213.190:44427 requested a full tablet report, sending...
I20240507 08:04:41.969944 9046 webserver.cc:460] Webserver started at http://127.8.213.130:36623/ using document root <none> and password file <none>
I20240507 08:04:41.970417 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:41.970600 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:41.970849 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:41.970834 9583 ts_manager.cc:194] Registered new tserver with Master: e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337)
I20240507 08:04:41.972093 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-1-root/instance:
uuid: "889812a87b2846a4979b1496ba962952"
format_stamp: "Formatted at 2024-05-07 08:04:41 on dist-test-slave-ls56"
I20240507 08:04:41.972597 9583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43014
I20240507 08:04:41.976572 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240507 08:04:41.979933 9722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:41.980839 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240507 08:04:41.981204 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-1-root
uuid: "889812a87b2846a4979b1496ba962952"
format_stamp: "Formatted at 2024-05-07 08:04:41 on dist-test-slave-ls56"
I20240507 08:04:41.981666 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:42.013223 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:42.014477 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:42.015870 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:04:42.018119 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:04:42.018294 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:42.018543 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:04:42.018683 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:42.056815 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.130:42723
I20240507 08:04:42.056910 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.130:42723 every 8 connection(s)
I20240507 08:04:42.061350 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:04:42.069270 9788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:04:42.071355 9789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:42.073567 9046 server_base.cc:1034] running on GCE node
I20240507 08:04:42.073873 9785 heartbeater.cc:347] Connected to a master server at 127.8.213.190:44427
I20240507 08:04:42.074366 9785 heartbeater.cc:464] Registering TS with master...
W20240507 08:04:42.074395 9791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:04:42.075357 9785 heartbeater.cc:511] Master 127.8.213.190:44427 requested a full tablet report, sending...
I20240507 08:04:42.075475 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:04:42.075737 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:04:42.075937 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069082075919 us; error 0 us; skew 500 ppm
I20240507 08:04:42.076555 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:04:42.077445 9583 ts_manager.cc:194] Registered new tserver with Master: 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:42.078998 9583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43026
I20240507 08:04:42.079301 9046 webserver.cc:460] Webserver started at http://127.8.213.131:39215/ using document root <none> and password file <none>
I20240507 08:04:42.079823 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:04:42.080013 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:04:42.080253 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:04:42.081382 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-2-root/instance:
uuid: "16d2a767951f4624a42a1ffb7ed2a342"
format_stamp: "Formatted at 2024-05-07 08:04:42 on dist-test-slave-ls56"
I20240507 08:04:42.085888 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240507 08:04:42.088835 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:42.089536 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240507 08:04:42.089835 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-2-root
uuid: "16d2a767951f4624a42a1ffb7ed2a342"
format_stamp: "Formatted at 2024-05-07 08:04:42 on dist-test-slave-ls56"
I20240507 08:04:42.090111 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1715069050956306-9046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:04:42.118449 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:04:42.119619 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:04:42.121032 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:04:42.123209 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:04:42.123416 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:42.123639 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:04:42.123786 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:04:42.160467 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.131:36017
I20240507 08:04:42.160619 9858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.131:36017 every 8 connection(s)
I20240507 08:04:42.174332 9859 heartbeater.cc:347] Connected to a master server at 127.8.213.190:44427
I20240507 08:04:42.174742 9859 heartbeater.cc:464] Registering TS with master...
I20240507 08:04:42.175406 9859 heartbeater.cc:511] Master 127.8.213.190:44427 requested a full tablet report, sending...
I20240507 08:04:42.177166 9583 ts_manager.cc:194] Registered new tserver with Master: 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.177938 9046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014400066s
I20240507 08:04:42.178411 9583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43036
I20240507 08:04:42.180662 9046 test_util.cc:268] Using random seed: -1553425374
I20240507 08:04:42.201769 9583 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43052:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:04:42.204533 9583 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:04:42.270562 9675 tablet_service.cc:1467] Processing CreateTablet for tablet ee5529e859954b9e822f87556c72c5df (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240507 08:04:42.271967 9675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5529e859954b9e822f87556c72c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.274597 9671 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1b3e789e14494b2d37fbc8409184e (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240507 08:04:42.274454 9673 tablet_service.cc:1467] Processing CreateTablet for tablet 2152d4c8e1a948f4bf423fb3cac7fc3c (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240507 08:04:42.275147 9672 tablet_service.cc:1467] Processing CreateTablet for tablet 7146f803bde54a5889435a67cfe34cab (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240507 08:04:42.275905 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1b3e789e14494b2d37fbc8409184e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.276530 9673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152d4c8e1a948f4bf423fb3cac7fc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.278381 9674 tablet_service.cc:1467] Processing CreateTablet for tablet 08be1fa1f2c6414bababd3c25c338d4e (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240507 08:04:42.280088 9674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08be1fa1f2c6414bababd3c25c338d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.283814 9672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7146f803bde54a5889435a67cfe34cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.285382 9670 tablet_service.cc:1467] Processing CreateTablet for tablet e9e5590e59cc48168ab470d065e905c4 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240507 08:04:42.286661 9670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e5590e59cc48168ab470d065e905c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.300755 9824 tablet_service.cc:1467] Processing CreateTablet for tablet ee5529e859954b9e822f87556c72c5df (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240507 08:04:42.302151 9824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5529e859954b9e822f87556c72c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.302857 9669 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d1cca6e264060bbf1182f137d9457 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240507 08:04:42.304157 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d1cca6e264060bbf1182f137d9457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.306766 9823 tablet_service.cc:1467] Processing CreateTablet for tablet 08be1fa1f2c6414bababd3c25c338d4e (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240507 08:04:42.308028 9823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08be1fa1f2c6414bababd3c25c338d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.308234 9668 tablet_service.cc:1467] Processing CreateTablet for tablet 0a02c28220e04250a8f9c673bef04ae6 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240507 08:04:42.308720 9666 tablet_service.cc:1467] Processing CreateTablet for tablet bdcca60a5c704607b61bce4d80e4d4b1 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240507 08:04:42.309590 9668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a02c28220e04250a8f9c673bef04ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.310915 9666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcca60a5c704607b61bce4d80e4d4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.310763 9822 tablet_service.cc:1467] Processing CreateTablet for tablet 2152d4c8e1a948f4bf423fb3cac7fc3c (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240507 08:04:42.312037 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152d4c8e1a948f4bf423fb3cac7fc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.314469 9820 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1b3e789e14494b2d37fbc8409184e (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240507 08:04:42.314848 9667 tablet_service.cc:1467] Processing CreateTablet for tablet bfd011611a764aa880282abfd309fccd (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240507 08:04:42.315806 9820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1b3e789e14494b2d37fbc8409184e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.316162 9667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd011611a764aa880282abfd309fccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.318763 9821 tablet_service.cc:1467] Processing CreateTablet for tablet 7146f803bde54a5889435a67cfe34cab (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240507 08:04:42.320047 9821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7146f803bde54a5889435a67cfe34cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.345099 9819 tablet_service.cc:1467] Processing CreateTablet for tablet e9e5590e59cc48168ab470d065e905c4 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240507 08:04:42.346463 9819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e5590e59cc48168ab470d065e905c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.350778 9818 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d1cca6e264060bbf1182f137d9457 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240507 08:04:42.352077 9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d1cca6e264060bbf1182f137d9457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.354823 9817 tablet_service.cc:1467] Processing CreateTablet for tablet 0a02c28220e04250a8f9c673bef04ae6 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240507 08:04:42.356120 9817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a02c28220e04250a8f9c673bef04ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.358829 9815 tablet_service.cc:1467] Processing CreateTablet for tablet bdcca60a5c704607b61bce4d80e4d4b1 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240507 08:04:42.359671 9750 tablet_service.cc:1467] Processing CreateTablet for tablet ee5529e859954b9e822f87556c72c5df (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION VALUES < 214748364
I20240507 08:04:42.360152 9815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcca60a5c704607b61bce4d80e4d4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.360904 9750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5529e859954b9e822f87556c72c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.362763 9816 tablet_service.cc:1467] Processing CreateTablet for tablet bfd011611a764aa880282abfd309fccd (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240507 08:04:42.362656 9744 tablet_service.cc:1467] Processing CreateTablet for tablet eb9d1cca6e264060bbf1182f137d9457 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20240507 08:04:42.364169 9816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd011611a764aa880282abfd309fccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.363898 9748 tablet_service.cc:1467] Processing CreateTablet for tablet 2152d4c8e1a948f4bf423fb3cac7fc3c (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20240507 08:04:42.364455 9744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9d1cca6e264060bbf1182f137d9457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.365214 9748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2152d4c8e1a948f4bf423fb3cac7fc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.366742 9747 tablet_service.cc:1467] Processing CreateTablet for tablet 7146f803bde54a5889435a67cfe34cab (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20240507 08:04:42.368038 9747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7146f803bde54a5889435a67cfe34cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.367712 9743 tablet_service.cc:1467] Processing CreateTablet for tablet 0a02c28220e04250a8f9c673bef04ae6 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20240507 08:04:42.369396 9743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a02c28220e04250a8f9c673bef04ae6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.370782 9746 tablet_service.cc:1467] Processing CreateTablet for tablet c3f1b3e789e14494b2d37fbc8409184e (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20240507 08:04:42.372058 9746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f1b3e789e14494b2d37fbc8409184e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.374965 9745 tablet_service.cc:1467] Processing CreateTablet for tablet e9e5590e59cc48168ab470d065e905c4 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20240507 08:04:42.375571 9742 tablet_service.cc:1467] Processing CreateTablet for tablet bfd011611a764aa880282abfd309fccd (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20240507 08:04:42.376494 9745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e5590e59cc48168ab470d065e905c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.377270 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd011611a764aa880282abfd309fccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.362763 9749 tablet_service.cc:1467] Processing CreateTablet for tablet 08be1fa1f2c6414bababd3c25c338d4e (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20240507 08:04:42.381143 9741 tablet_service.cc:1467] Processing CreateTablet for tablet bdcca60a5c704607b61bce4d80e4d4b1 (DEFAULT_TABLE table=test-workload [id=c9b9f35019b64d03b05ada083017e62a]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20240507 08:04:42.389957 9741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdcca60a5c704607b61bce4d80e4d4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.391345 9749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08be1fa1f2c6414bababd3c25c338d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:04:42.446610 9874 tablet_bootstrap.cc:492] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.452940 9874 tablet_bootstrap.cc:654] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.470072 9875 tablet_bootstrap.cc:492] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.482582 9875 tablet_bootstrap.cc:654] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.484742 9876 tablet_bootstrap.cc:492] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.494383 9874 tablet_bootstrap.cc:492] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.494874 9874 ts_tablet_manager.cc:1367] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.012s
I20240507 08:04:42.497519 9874 raft_consensus.cc:348] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.498257 9874 raft_consensus.cc:374] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.498569 9874 raft_consensus.cc:729] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.499222 9874 consensus_queue.cc:260] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.499876 9876 tablet_bootstrap.cc:654] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.502967 9874 ts_tablet_manager.cc:1398] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20240507 08:04:42.506567 9710 heartbeater.cc:503] Master 127.8.213.190:44427 was elected leader, sending a full tablet report...
I20240507 08:04:42.506783 9875 tablet_bootstrap.cc:492] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.507920 9874 tablet_bootstrap.cc:492] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.509652 9875 ts_tablet_manager.cc:1367] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.006s
I20240507 08:04:42.511758 9876 tablet_bootstrap.cc:492] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:42.512241 9876 ts_tablet_manager.cc:1367] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.008s
I20240507 08:04:42.512401 9875 raft_consensus.cc:348] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.513101 9874 tablet_bootstrap.cc:654] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.513213 9875 raft_consensus.cc:374] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.513603 9875 raft_consensus.cc:729] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.514465 9875 consensus_queue.cc:260] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.515162 9876 raft_consensus.cc:348] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.516081 9876 raft_consensus.cc:374] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.516512 9876 raft_consensus.cc:729] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.517558 9876 consensus_queue.cc:260] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.523938 9875 ts_tablet_manager.cc:1398] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20240507 08:04:42.525823 9859 heartbeater.cc:503] Master 127.8.213.190:44427 was elected leader, sending a full tablet report...
I20240507 08:04:42.527590 9874 tablet_bootstrap.cc:492] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.528209 9874 ts_tablet_manager.cc:1367] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.016s
I20240507 08:04:42.529628 9875 tablet_bootstrap.cc:492] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.531801 9874 raft_consensus.cc:348] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.532793 9874 raft_consensus.cc:374] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.533164 9874 raft_consensus.cc:729] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.534085 9874 consensus_queue.cc:260] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.537350 9874 ts_tablet_manager.cc:1398] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.009s user 0.007s sys 0.003s
I20240507 08:04:42.538523 9874 tablet_bootstrap.cc:492] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.542495 9785 heartbeater.cc:503] Master 127.8.213.190:44427 was elected leader, sending a full tablet report...
I20240507 08:04:42.543706 9876 ts_tablet_manager.cc:1398] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.031s user 0.000s sys 0.029s
I20240507 08:04:42.544147 9875 tablet_bootstrap.cc:654] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.544809 9876 tablet_bootstrap.cc:492] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.545163 9874 tablet_bootstrap.cc:654] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.552466 9876 tablet_bootstrap.cc:654] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.601158 9875 tablet_bootstrap.cc:492] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.601759 9875 ts_tablet_manager.cc:1367] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.000s
I20240507 08:04:42.604388 9875 raft_consensus.cc:348] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.605149 9875 raft_consensus.cc:374] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.605439 9875 raft_consensus.cc:729] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.606660 9876 tablet_bootstrap.cc:492] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:42.606292 9875 consensus_queue.cc:260] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.607276 9876 ts_tablet_manager.cc:1367] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.046s
I20240507 08:04:42.608225 9875 ts_tablet_manager.cc:1398] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:04:42.608667 9874 tablet_bootstrap.cc:492] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.609004 9875 tablet_bootstrap.cc:492] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.609127 9874 ts_tablet_manager.cc:1367] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.071s user 0.033s sys 0.027s
I20240507 08:04:42.609998 9876 raft_consensus.cc:348] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.610729 9876 raft_consensus.cc:374] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.611020 9876 raft_consensus.cc:729] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.611776 9874 raft_consensus.cc:348] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.611723 9876 consensus_queue.cc:260] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.612493 9874 raft_consensus.cc:374] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.612802 9874 raft_consensus.cc:729] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.613472 9875 tablet_bootstrap.cc:654] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.613976 9876 ts_tablet_manager.cc:1398] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240507 08:04:42.613548 9874 consensus_queue.cc:260] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.614842 9876 tablet_bootstrap.cc:492] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.616088 9874 ts_tablet_manager.cc:1398] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240507 08:04:42.617054 9874 tablet_bootstrap.cc:492] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.621308 9876 tablet_bootstrap.cc:654] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.622864 9874 tablet_bootstrap.cc:654] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.631850 9876 tablet_bootstrap.cc:492] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:42.632411 9876 ts_tablet_manager.cc:1367] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.005s
I20240507 08:04:42.634411 9875 tablet_bootstrap.cc:492] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.634626 9874 tablet_bootstrap.cc:492] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.634878 9875 ts_tablet_manager.cc:1367] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.000s
I20240507 08:04:42.635105 9874 ts_tablet_manager.cc:1367] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20240507 08:04:42.635105 9876 raft_consensus.cc:348] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.635741 9878 raft_consensus.cc:482] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.636044 9876 raft_consensus.cc:374] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.636364 9876 raft_consensus.cc:729] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.636204 9878 raft_consensus.cc:504] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.637077 9876 consensus_queue.cc:260] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.638329 9874 raft_consensus.cc:348] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.639110 9874 raft_consensus.cc:374] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.639326 9876 ts_tablet_manager.cc:1398] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240507 08:04:42.639456 9874 raft_consensus.cc:729] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.640326 9876 tablet_bootstrap.cc:492] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.640306 9874 consensus_queue.cc:260] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.647473 9876 tablet_bootstrap.cc:654] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.647506 9878 leader_election.cc:288] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.652567 9874 ts_tablet_manager.cc:1398] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20240507 08:04:42.653645 9874 tablet_bootstrap.cc:492] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.637470 9875 raft_consensus.cc:348] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.656327 9875 raft_consensus.cc:374] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.656661 9875 raft_consensus.cc:729] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.657603 9875 consensus_queue.cc:260] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.659700 9874 tablet_bootstrap.cc:654] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.662086 9875 ts_tablet_manager.cc:1398] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.027s user 0.005s sys 0.002s
I20240507 08:04:42.663162 9875 tablet_bootstrap.cc:492] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.669800 9882 raft_consensus.cc:482] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.670297 9882 raft_consensus.cc:504] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.671525 9874 tablet_bootstrap.cc:492] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.672044 9874 ts_tablet_manager.cc:1367] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20240507 08:04:42.672864 9882 leader_election.cc:288] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:42.693634 9892 raft_consensus.cc:482] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.694422 9883 raft_consensus.cc:482] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.694360 9892 raft_consensus.cc:504] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.695122 9883 raft_consensus.cc:504] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.697046 9892 leader_election.cc:288] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.676116 9874 raft_consensus.cc:348] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.718788 9876 tablet_bootstrap.cc:492] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:42.719295 9876 ts_tablet_manager.cc:1367] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.032s
I20240507 08:04:42.719559 9875 tablet_bootstrap.cc:654] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.715783 9874 raft_consensus.cc:374] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.720399 9874 raft_consensus.cc:729] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.721130 9874 consensus_queue.cc:260] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.729938 9874 ts_tablet_manager.cc:1398] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.058s user 0.006s sys 0.001s
I20240507 08:04:42.730896 9874 tablet_bootstrap.cc:492] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.717213 9883 leader_election.cc:288] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.734066 9883 raft_consensus.cc:482] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.734494 9883 raft_consensus.cc:504] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.736418 9883 leader_election.cc:288] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.738219 9876 raft_consensus.cc:348] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.741689 9876 raft_consensus.cc:374] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.742482 9876 raft_consensus.cc:729] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.744799 9876 consensus_queue.cc:260] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.746230 9874 tablet_bootstrap.cc:654] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.752506 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.753322 9834 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 0.
I20240507 08:04:42.754549 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.755522 9892 raft_consensus.cc:482] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.755954 9892 raft_consensus.cc:504] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.756538 9876 ts_tablet_manager.cc:1398] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.037s user 0.004s sys 0.005s
I20240507 08:04:42.757875 9892 leader_election.cc:288] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.758013 9876 tablet_bootstrap.cc:492] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.769333 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:42.769855 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1b3e789e14494b2d37fbc8409184e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.770071 9759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:42.770534 9833 raft_consensus.cc:2454] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 0.
I20240507 08:04:42.770440 9760 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 0.
I20240507 08:04:42.769446 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:42.774076 9685 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 0.
I20240507 08:04:42.775552 9800 leader_election.cc:302] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:42.776335 9875 tablet_bootstrap.cc:492] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.776469 9882 raft_consensus.cc:2781] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:42.776808 9882 raft_consensus.cc:482] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:42.777123 9882 raft_consensus.cc:3037] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.780047 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1b3e789e14494b2d37fbc8409184e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
W20240507 08:04:42.784468 9650 leader_election.cc:341] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:04:42.784768 9631 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:04:42.786592 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:42.787194 9760 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 0.
I20240507 08:04:42.785230 9649 leader_election.cc:302] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:42.790484 9892 raft_consensus.cc:2781] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:42.790877 9892 raft_consensus.cc:482] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:42.791232 9892 raft_consensus.cc:3037] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.792822 9876 tablet_bootstrap.cc:654] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
W20240507 08:04:42.794360 9650 leader_election.cc:341] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.789839 9875 ts_tablet_manager.cc:1367] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.127s user 0.016s sys 0.012s
I20240507 08:04:42.795426 9882 raft_consensus.cc:504] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.798728 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:42.798990 9892 raft_consensus.cc:504] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.800119 9685 raft_consensus.cc:2379] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 16d2a767951f4624a42a1ffb7ed2a342 in current term 1: Already voted for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in this term.
W20240507 08:04:42.801779 9649 leader_election.cc:341] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.802203 9649 leader_election.cc:302] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952
I20240507 08:04:42.802311 9892 leader_election.cc:288] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.802374 9882 leader_election.cc:288] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:42.803009 9878 raft_consensus.cc:2726] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:42.803288 9649 leader_election.cc:302] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 889812a87b2846a4979b1496ba962952
I20240507 08:04:42.803447 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:42.803984 9878 raft_consensus.cc:2781] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:42.804003 9760 raft_consensus.cc:3037] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.804420 9878 raft_consensus.cc:482] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:42.804698 9878 raft_consensus.cc:3037] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.805104 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:42.805899 9833 raft_consensus.cc:2379] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in current term 1: Already voted for candidate 16d2a767951f4624a42a1ffb7ed2a342 in this term.
I20240507 08:04:42.805929 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:42.806823 9685 raft_consensus.cc:2379] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 889812a87b2846a4979b1496ba962952 in current term 1: Already voted for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in this term.
I20240507 08:04:42.807861 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.798211 9875 raft_consensus.cc:348] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.808414 9875 raft_consensus.cc:374] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.808611 9833 raft_consensus.cc:2379] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 889812a87b2846a4979b1496ba962952 in current term 1: Already voted for candidate 16d2a767951f4624a42a1ffb7ed2a342 in this term.
I20240507 08:04:42.808714 9875 raft_consensus.cc:729] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.809546 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.809535 9875 consensus_queue.cc:260] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.811236 9760 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 1.
I20240507 08:04:42.811902 9875 ts_tablet_manager.cc:1398] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20240507 08:04:42.811586 9878 raft_consensus.cc:504] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.812875 9875 tablet_bootstrap.cc:492] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.813401 9799 leader_election.cc:302] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters: e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:42.813544 9878 leader_election.cc:288] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.814356 9882 raft_consensus.cc:2781] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:42.814471 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:42.818912 9874 tablet_bootstrap.cc:492] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.818369 9726 leader_election.cc:302] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 889812a87b2846a4979b1496ba962952; no voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64
W20240507 08:04:42.817799 9724 leader_election.cc:341] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.823845 9874 ts_tablet_manager.cc:1367] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.093s user 0.041s sys 0.009s
I20240507 08:04:42.824012 9899 raft_consensus.cc:2726] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:04:42.825237 9726 leader_election.cc:341] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.825696 9726 leader_election.cc:302] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 889812a87b2846a4979b1496ba962952; no voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:42.826421 9899 raft_consensus.cc:2726] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:42.817687 9759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1b3e789e14494b2d37fbc8409184e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:42.827046 9874 raft_consensus.cc:348] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.827981 9874 raft_consensus.cc:374] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.818789 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f1b3e789e14494b2d37fbc8409184e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
W20240507 08:04:42.828287 9650 leader_election.cc:341] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.828568 9834 raft_consensus.cc:3037] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.829835 9875 tablet_bootstrap.cc:654] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.831730 9876 tablet_bootstrap.cc:492] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:42.832209 9876 ts_tablet_manager.cc:1367] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.000s
I20240507 08:04:42.835017 9876 raft_consensus.cc:348] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.835578 9876 raft_consensus.cc:374] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.835804 9876 raft_consensus.cc:729] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.836272 9834 raft_consensus.cc:2454] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 1.
I20240507 08:04:42.836563 9875 tablet_bootstrap.cc:492] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.836366 9876 consensus_queue.cc:260] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.818647 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:42.837010 9875 ts_tablet_manager.cc:1367] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.009s
I20240507 08:04:42.837538 9760 raft_consensus.cc:2379] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in current term 1: Already voted for candidate 16d2a767951f4624a42a1ffb7ed2a342 in this term.
I20240507 08:04:42.837539 9649 leader_election.cc:302] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 889812a87b2846a4979b1496ba962952
I20240507 08:04:42.837066 9874 raft_consensus.cc:729] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.838549 9650 leader_election.cc:302] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952
I20240507 08:04:42.838932 9878 raft_consensus.cc:2781] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:42.839633 9892 raft_consensus.cc:2726] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:04:42.839252 9874 consensus_queue.cc:260] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.841609 9874 ts_tablet_manager.cc:1398] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.017s user 0.003s sys 0.002s
I20240507 08:04:42.842554 9874 tablet_bootstrap.cc:492] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.842861 9875 raft_consensus.cc:348] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.843600 9875 raft_consensus.cc:374] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.843890 9875 raft_consensus.cc:729] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.844610 9875 consensus_queue.cc:260] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.847608 9878 raft_consensus.cc:686] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:42.848241 9876 ts_tablet_manager.cc:1398] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20240507 08:04:42.848623 9878 consensus_queue.cc:237] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.849663 9874 tablet_bootstrap.cc:654] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.854868 9875 ts_tablet_manager.cc:1398] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20240507 08:04:42.856940 9876 tablet_bootstrap.cc:492] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.856993 9875 tablet_bootstrap.cc:492] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.862241 9874 tablet_bootstrap.cc:492] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.862740 9874 ts_tablet_manager.cc:1367] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.008s
I20240507 08:04:42.867986 9875 tablet_bootstrap.cc:654] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.866216 9882 raft_consensus.cc:686] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:42.868005 9874 raft_consensus.cc:348] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.869015 9874 raft_consensus.cc:374] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.869400 9874 raft_consensus.cc:729] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.869208 9882 consensus_queue.cc:237] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.870316 9874 consensus_queue.cc:260] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.873371 9874 ts_tablet_manager.cc:1398] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20240507 08:04:42.874807 9874 tablet_bootstrap.cc:492] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.875284 9876 tablet_bootstrap.cc:654] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.873237 9582 catalog_manager.cc:5484] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 1 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:42.880542 9899 raft_consensus.cc:482] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.881202 9899 raft_consensus.cc:504] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.884721 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd011611a764aa880282abfd309fccd" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:42.885454 9685 raft_consensus.cc:2454] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 0.
I20240507 08:04:42.885131 9581 catalog_manager.cc:5484] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:42.886236 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd011611a764aa880282abfd309fccd" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.886705 9726 leader_election.cc:302] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:42.887581 9883 raft_consensus.cc:2781] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:42.887953 9883 raft_consensus.cc:482] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:42.888268 9883 raft_consensus.cc:3037] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:04:42.888371 9724 leader_election.cc:341] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.889147 9899 leader_election.cc:288] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.892047 9874 tablet_bootstrap.cc:654] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.892285 9875 tablet_bootstrap.cc:492] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.892773 9875 ts_tablet_manager.cc:1367] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.016s
I20240507 08:04:42.895529 9875 raft_consensus.cc:348] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.896317 9875 raft_consensus.cc:374] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.896631 9875 raft_consensus.cc:729] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.896445 9883 raft_consensus.cc:504] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.897378 9875 consensus_queue.cc:260] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.899338 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd011611a764aa880282abfd309fccd" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:42.899945 9685 raft_consensus.cc:3037] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.900024 9875 ts_tablet_manager.cc:1398] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240507 08:04:42.900472 9874 tablet_bootstrap.cc:492] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.901072 9875 tablet_bootstrap.cc:492] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.901113 9874 ts_tablet_manager.cc:1367] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.007s
I20240507 08:04:42.904024 9874 raft_consensus.cc:348] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.904704 9876 tablet_bootstrap.cc:492] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:42.904812 9874 raft_consensus.cc:374] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.905223 9874 raft_consensus.cc:729] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.906097 9874 consensus_queue.cc:260] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.907322 9685 raft_consensus.cc:2454] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 1.
I20240507 08:04:42.907552 9875 tablet_bootstrap.cc:654] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.905229 9876 ts_tablet_manager.cc:1367] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.009s
I20240507 08:04:42.908785 9874 ts_tablet_manager.cc:1398] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240507 08:04:42.908588 9726 leader_election.cc:302] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:42.909662 9899 raft_consensus.cc:2781] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:42.909821 9874 tablet_bootstrap.cc:492] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:42.917685 9883 leader_election.cc:288] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.919114 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd011611a764aa880282abfd309fccd" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:42.919762 9876 raft_consensus.cc:348] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.920522 9876 raft_consensus.cc:374] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:04:42.920686 9724 leader_election.cc:341] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.920840 9876 raft_consensus.cc:729] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.921512 9876 consensus_queue.cc:260] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.922161 9899 raft_consensus.cc:686] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Leader. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:42.923094 9899 consensus_queue.cc:237] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.924733 9874 tablet_bootstrap.cc:654] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.926045 9876 ts_tablet_manager.cc:1398] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20240507 08:04:42.927584 9876 tablet_bootstrap.cc:492] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:42.929328 9878 raft_consensus.cc:482] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.930795 9875 tablet_bootstrap.cc:492] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:42.931550 9875 ts_tablet_manager.cc:1367] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.010s
I20240507 08:04:42.934355 9907 raft_consensus.cc:482] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.934836 9907 raft_consensus.cc:504] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.935380 9875 raft_consensus.cc:348] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.936640 9875 raft_consensus.cc:374] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.937136 9907 leader_election.cc:288] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.937089 9875 raft_consensus.cc:729] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.938390 9875 consensus_queue.cc:260] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.940850 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.941495 9834 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 0.
I20240507 08:04:42.942425 9875 ts_tablet_manager.cc:1398] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20240507 08:04:42.942631 9649 leader_election.cc:302] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:42.939154 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:42.943620 9760 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 0.
I20240507 08:04:42.943802 9875 tablet_bootstrap.cc:492] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:42.929826 9878 raft_consensus.cc:504] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.946954 9878 leader_election.cc:288] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.947474 9878 raft_consensus.cc:2781] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:42.947813 9878 raft_consensus.cc:482] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:42.948143 9878 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.950706 9875 tablet_bootstrap.cc:654] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.951436 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.952154 9834 raft_consensus.cc:2454] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 0.
I20240507 08:04:42.953650 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
W20240507 08:04:42.955051 9650 leader_election.cc:341] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:42.955832 9649 leader_election.cc:302] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 889812a87b2846a4979b1496ba962952
I20240507 08:04:42.956725 9892 raft_consensus.cc:2781] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:42.957197 9892 raft_consensus.cc:482] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:42.957641 9892 raft_consensus.cc:3037] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.962800 9878 raft_consensus.cc:504] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.964929 9876 tablet_bootstrap.cc:654] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:42.967624 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:42.968407 9760 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:42.971892 9874 tablet_bootstrap.cc:492] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:42.972474 9874 ts_tablet_manager.cc:1367] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.000s
I20240507 08:04:42.975788 9892 raft_consensus.cc:504] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.975899 9874 raft_consensus.cc:348] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.976920 9874 raft_consensus.cc:374] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:42.977274 9874 raft_consensus.cc:729] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:42.977917 9882 raft_consensus.cc:482] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.978106 9874 consensus_queue.cc:260] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:42.978829 9882 raft_consensus.cc:504] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.976756 9581 catalog_manager.cc:5484] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889812a87b2846a4979b1496ba962952 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "889812a87b2846a4979b1496ba962952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:42.980801 9874 ts_tablet_manager.cc:1398] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20240507 08:04:42.978273 9878 leader_election.cc:288] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:42.993373 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:42.996467 9899 raft_consensus.cc:482] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.986243 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:43.040388 9874 tablet_bootstrap.cc:492] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64: Bootstrap starting.
I20240507 08:04:43.040781 9833 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.040447 9899 raft_consensus.cc:504] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:42.987773 9759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:43.042605 9899 leader_election.cc:288] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.043706 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.044178 9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:42.980791 9882 leader_election.cc:288] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:43.033903 9907 raft_consensus.cc:482] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.989851 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:42.981213 9892 leader_election.cc:288] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.053079 9874 tablet_bootstrap.cc:654] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:43.013998 9917 raft_consensus.cc:482] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:42.985726 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
W20240507 08:04:43.047081 9724 leader_election.cc:341] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.044459 9684 raft_consensus.cc:2454] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 0.
W20240507 08:04:43.045220 9799 leader_election.cc:341] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.064953 9907 raft_consensus.cc:504] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.069120 9685 raft_consensus.cc:2454] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 0.
I20240507 08:04:43.067915 9917 raft_consensus.cc:504] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.072021 9917 leader_election.cc:288] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.072734 9726 leader_election.cc:302] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 16d2a767951f4624a42a1ffb7ed2a342
I20240507 08:04:43.073007 9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:43.073705 9832 raft_consensus.cc:2454] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 0.
I20240507 08:04:43.073658 9800 leader_election.cc:302] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 889812a87b2846a4979b1496ba962952
I20240507 08:04:43.072968 9876 tablet_bootstrap.cc:492] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:43.075611 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.075773 9905 raft_consensus.cc:2781] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:43.075992 9876 ts_tablet_manager.cc:1367] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.000s
I20240507 08:04:43.076198 9905 raft_consensus.cc:482] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:43.076603 9905 raft_consensus.cc:3037] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.077936 9832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:43.078290 9907 leader_election.cc:288] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.079311 9876 raft_consensus.cc:348] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.079826 9876 raft_consensus.cc:374] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.080050 9876 raft_consensus.cc:729] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.080614 9876 consensus_queue.cc:260] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
W20240507 08:04:43.075062 9650 leader_election.cc:341] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.082000 9876 ts_tablet_manager.cc:1398] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:04:43.083433 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:43.084210 9876 tablet_bootstrap.cc:492] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:43.084357 9917 raft_consensus.cc:2781] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240507 08:04:43.084625 9650 leader_election.cc:341] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.084750 9917 raft_consensus.cc:482] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:43.085157 9917 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.070123 9882 raft_consensus.cc:482] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:43.086336 9882 raft_consensus.cc:504] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:43.067842 9834 raft_consensus.cc:3037] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.086583 9905 raft_consensus.cc:504] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.075057 9724 leader_election.cc:302] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:04:43.093508 9876 tablet_bootstrap.cc:654] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:43.088743 9882 leader_election.cc:288] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:43.089504 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.094437 9899 raft_consensus.cc:2781] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:43.094733 9685 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 0.
W20240507 08:04:43.093641 9726 leader_election.cc:341] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.091089 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:43.095449 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:43.093881 9832 raft_consensus.cc:2379] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in current term 1: Already voted for candidate 16d2a767951f4624a42a1ffb7ed2a342 in this term.
I20240507 08:04:43.093014 9760 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 1.
I20240507 08:04:43.094810 9899 raft_consensus.cc:482] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:43.095937 9800 leader_election.cc:302] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:43.092896 9917 raft_consensus.cc:504] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.095806 9758 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 0.
I20240507 08:04:43.096526 9899 raft_consensus.cc:3037] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.087929 9875 tablet_bootstrap.cc:492] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:43.097432 9650 leader_election.cc:302] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:43.096035 9684 raft_consensus.cc:3037] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.097996 9908 raft_consensus.cc:2781] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:43.100006 9875 ts_tablet_manager.cc:1367] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.156s user 0.022s sys 0.042s
I20240507 08:04:43.102564 9833 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 1.
I20240507 08:04:43.099612 9649 leader_election.cc:302] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952
I20240507 08:04:43.104506 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:43.102993 9908 raft_consensus.cc:482] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:43.105343 9908 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.100569 9907 raft_consensus.cc:2781] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:04:43.105769 9799 leader_election.cc:341] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.105633 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:43.105922 9917 leader_election.cc:288] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.108191 9685 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.108947 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:43.118222 9834 raft_consensus.cc:2454] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 1.
W20240507 08:04:43.119158 9724 leader_election.cc:341] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.119961 9685 raft_consensus.cc:2454] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 1.
I20240507 08:04:43.121080 9726 leader_election.cc:302] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 16d2a767951f4624a42a1ffb7ed2a342
I20240507 08:04:43.121100 9649 leader_election.cc:302] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 889812a87b2846a4979b1496ba962952
I20240507 08:04:43.122578 9875 raft_consensus.cc:348] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.122896 9878 raft_consensus.cc:2781] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:43.110667 9907 raft_consensus.cc:686] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:43.123224 9917 raft_consensus.cc:2781] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:43.123721 9917 raft_consensus.cc:686] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Leader. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:43.123993 9907 consensus_queue.cc:237] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.124485 9917 consensus_queue.cc:237] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.123247 9875 raft_consensus.cc:374] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.126359 9875 raft_consensus.cc:729] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.106107 9905 leader_election.cc:288] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:43.127082 9875 consensus_queue.cc:260] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.130079 9875 ts_tablet_manager.cc:1398] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20240507 08:04:43.131282 9875 tablet_bootstrap.cc:492] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:43.133802 9899 raft_consensus.cc:504] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.142570 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
W20240507 08:04:43.148000 9726 leader_election.cc:341] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.150606 9684 raft_consensus.cc:2454] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 1.
I20240507 08:04:43.151335 9892 raft_consensus.cc:2726] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:43.154552 9875 tablet_bootstrap.cc:654] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:43.146277 9908 raft_consensus.cc:504] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:43.160161 9800 leader_election.cc:302] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters: 889812a87b2846a4979b1496ba962952
I20240507 08:04:43.164100 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:43.161950 9905 raft_consensus.cc:2781] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:43.164736 9758 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.164824 9905 raft_consensus.cc:686] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:43.164222 9876 tablet_bootstrap.cc:492] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:43.166002 9876 ts_tablet_manager.cc:1367] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.082s user 0.023s sys 0.008s
I20240507 08:04:43.165798 9905 consensus_queue.cc:237] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.160214 9899 leader_election.cc:288] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.171073 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:43.171666 9834 raft_consensus.cc:3037] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.172124 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:43.172708 9684 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.162050 9878 raft_consensus.cc:686] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:43.168560 9908 leader_election.cc:288] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:43.181470 9834 raft_consensus.cc:2454] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 1.
I20240507 08:04:43.182252 9684 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 1.
I20240507 08:04:43.182627 9876 raft_consensus.cc:348] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.183462 9876 raft_consensus.cc:374] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.184312 9876 raft_consensus.cc:729] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.185362 9876 consensus_queue.cc:260] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.183077 9724 leader_election.cc:302] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters: e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:43.167965 9582 catalog_manager.cc:5484] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 1 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.183409 9800 leader_election.cc:302] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:43.187081 9899 raft_consensus.cc:2781] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:43.187554 9899 raft_consensus.cc:686] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Leader. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:43.188313 9899 consensus_queue.cc:237] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.191953 9905 raft_consensus.cc:2781] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:43.192474 9905 raft_consensus.cc:686] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:43.192713 9581 catalog_manager.cc:5484] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:43.193492 9905 consensus_queue.cc:237] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:43.174024 9878 consensus_queue.cc:237] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.203441 9874 tablet_bootstrap.cc:492] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64: No bootstrap required, opened a new log
I20240507 08:04:43.204066 9874 ts_tablet_manager.cc:1367] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent bootstrapping tablet: real 0.164s user 0.017s sys 0.014s
I20240507 08:04:43.204708 9876 ts_tablet_manager.cc:1398] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.038s user 0.005s sys 0.001s
I20240507 08:04:43.189137 9758 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 1.
I20240507 08:04:43.206565 9876 tablet_bootstrap.cc:492] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:43.216264 9874 raft_consensus.cc:348] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.217281 9874 raft_consensus.cc:374] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.217656 9874 raft_consensus.cc:729] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.218492 9874 consensus_queue.cc:260] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.222081 9874 ts_tablet_manager.cc:1398] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64: Time spent starting tablet: real 0.018s user 0.006s sys 0.002s
I20240507 08:04:43.224608 9875 tablet_bootstrap.cc:492] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:43.223943 9583 catalog_manager.cc:5484] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889812a87b2846a4979b1496ba962952 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "889812a87b2846a4979b1496ba962952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.225039 9875 ts_tablet_manager.cc:1367] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.094s user 0.013s sys 0.001s
I20240507 08:04:43.225486 9876 tablet_bootstrap.cc:654] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:43.227660 9875 raft_consensus.cc:348] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.228377 9875 raft_consensus.cc:374] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.228642 9875 raft_consensus.cc:729] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.238346 9581 catalog_manager.cc:5484] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.240495 9875 consensus_queue.cc:260] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.242652 9875 ts_tablet_manager.cc:1398] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.017s user 0.003s sys 0.004s
I20240507 08:04:43.243068 9876 tablet_bootstrap.cc:492] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:43.243492 9876 ts_tablet_manager.cc:1367] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.004s
I20240507 08:04:43.243547 9875 tablet_bootstrap.cc:492] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342: Bootstrap starting.
I20240507 08:04:43.237638 9582 catalog_manager.cc:5484] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 1 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.245999 9876 raft_consensus.cc:348] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.246626 9876 raft_consensus.cc:374] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.246896 9876 raft_consensus.cc:729] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.247520 9876 consensus_queue.cc:260] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.249754 9876 ts_tablet_manager.cc:1398] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240507 08:04:43.250558 9876 tablet_bootstrap.cc:492] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952: Bootstrap starting.
I20240507 08:04:43.250970 9875 tablet_bootstrap.cc:654] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:43.253870 9899 raft_consensus.cc:482] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:43.254350 9899 raft_consensus.cc:504] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.256191 9876 tablet_bootstrap.cc:654] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952: Neither blocks nor log segments found. Creating new log.
I20240507 08:04:43.256270 9899 leader_election.cc:288] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.258905 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:43.260118 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.260797 9684 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 0.
I20240507 08:04:43.261811 9726 leader_election.cc:302] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
W20240507 08:04:43.262292 9724 leader_election.cc:341] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:04:43.262545 9899 raft_consensus.cc:2781] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:04:43.262889 9899 raft_consensus.cc:482] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:04:43.263257 9899 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.263042 9580 catalog_manager.cc:5484] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889812a87b2846a4979b1496ba962952 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "889812a87b2846a4979b1496ba962952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.269752 9876 tablet_bootstrap.cc:492] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952: No bootstrap required, opened a new log
I20240507 08:04:43.270036 9875 tablet_bootstrap.cc:492] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342: No bootstrap required, opened a new log
I20240507 08:04:43.270195 9876 ts_tablet_manager.cc:1367] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.004s
I20240507 08:04:43.270442 9875 ts_tablet_manager.cc:1367] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.006s
I20240507 08:04:43.272501 9875 raft_consensus.cc:348] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.272661 9876 raft_consensus.cc:348] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.273190 9875 raft_consensus.cc:374] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.273272 9876 raft_consensus.cc:374] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:04:43.273507 9875 raft_consensus.cc:729] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.273638 9876 raft_consensus.cc:729] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Initialized, Role: FOLLOWER
I20240507 08:04:43.274266 9876 consensus_queue.cc:260] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.274266 9875 consensus_queue.cc:260] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.276369 9876 ts_tablet_manager.cc:1398] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:04:43.276670 9875 ts_tablet_manager.cc:1398] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240507 08:04:43.280393 9878 consensus_queue.cc:1035] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.288558 9905 consensus_queue.cc:1035] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.269263 9899 raft_consensus.cc:504] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.293589 9905 raft_consensus.cc:482] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:43.295590 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:43.296084 9899 leader_election.cc:288] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.296299 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:43.296397 9684 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.296878 9833 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.294234 9905 raft_consensus.cc:504] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.305505 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.306176 9833 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 1.
I20240507 08:04:43.306644 9685 raft_consensus.cc:2379] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16d2a767951f4624a42a1ffb7ed2a342 in current term 1: Already voted for candidate 889812a87b2846a4979b1496ba962952 in this term.
I20240507 08:04:43.307715 9724 leader_election.cc:302] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:04:43.308862 9899 raft_consensus.cc:2781] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:04:43.309424 9899 raft_consensus.cc:686] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Leader. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:43.310195 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "889812a87b2846a4979b1496ba962952" is_pre_election: true
I20240507 08:04:43.310484 9899 consensus_queue.cc:237] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.312258 9799 leader_election.cc:302] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342; no voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:43.313803 9908 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.323125 9905 leader_election.cc:288] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:43.326409 9908 raft_consensus.cc:2726] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:43.329048 9684 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 1.
I20240507 08:04:43.342237 9580 catalog_manager.cc:5484] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 reported cstate change: term changed from 0 to 1, leader changed from <none> to 889812a87b2846a4979b1496ba962952 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "889812a87b2846a4979b1496ba962952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.344246 9760 raft_consensus.cc:3037] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.352265 9908 consensus_queue.cc:1035] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:43.353691 9907 consensus_queue.cc:1035] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.373342 9899 consensus_queue.cc:1035] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:43.388587 9834 raft_consensus.cc:3037] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.396577 9883 consensus_queue.cc:1035] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.478440 9834 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:58330
I20240507 08:04:43.479105 9834 raft_consensus.cc:595] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:43.486831 9834 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:58330
I20240507 08:04:43.487294 9834 raft_consensus.cc:595] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:43.530089 9883 raft_consensus.cc:482] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:43.530563 9883 raft_consensus.cc:504] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.532114 9883 leader_election.cc:288] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.533075 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.533139 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdcca60a5c704607b61bce4d80e4d4b1" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:43.533771 9685 raft_consensus.cc:2379] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 889812a87b2846a4979b1496ba962952 in current term 1: Already voted for candidate 16d2a767951f4624a42a1ffb7ed2a342 in this term.
I20240507 08:04:43.534610 9726 leader_election.cc:302] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 889812a87b2846a4979b1496ba962952; no voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:43.535221 9883 raft_consensus.cc:3037] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.539736 9883 raft_consensus.cc:2726] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:43.545782 9883 raft_consensus.cc:482] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:04:43.546217 9883 raft_consensus.cc:504] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.547740 9883 leader_election.cc:288] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.548558 9685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" is_pre_election: true
I20240507 08:04:43.548640 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342" is_pre_election: true
I20240507 08:04:43.549252 9834 raft_consensus.cc:2379] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 889812a87b2846a4979b1496ba962952 in current term 1: Already voted for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in this term.
I20240507 08:04:43.550091 9724 leader_election.cc:302] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 889812a87b2846a4979b1496ba962952; no voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:43.550701 9883 raft_consensus.cc:3037] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.555084 9883 raft_consensus.cc:2726] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:04:43.569900 9883 consensus_queue.cc:1035] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.580267 9883 consensus_queue.cc:1035] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.615988 9883 consensus_queue.cc:1035] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.627167 9685 raft_consensus.cc:3037] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:04:43.634421 9917 consensus_queue.cc:1035] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.663041 9908 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.672838 9882 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.681046 9905 raft_consensus.cc:979] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:43.681504 9905 raft_consensus.cc:1067] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:43.683145 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44186
I20240507 08:04:43.683727 9684 raft_consensus.cc:482] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:43.684062 9684 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:43.685005 9907 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.690351 9684 raft_consensus.cc:504] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:43.695420 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:43.696059 9833 raft_consensus.cc:3032] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:43.696359 9878 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.696835 9684 leader_election.cc:288] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Requested vote from peers 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:43.698971 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:43.699543 9760 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:43.696401 9833 raft_consensus.cc:729] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:43.702252 9833 consensus_queue.cc:260] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:43.703545 9833 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:43.706399 9907 consensus_queue.cc:1035] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:04:43.712795 9760 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:43.713924 9650 leader_election.cc:302] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:43.714879 9907 raft_consensus.cc:2781] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:43.715097 9833 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:43.715256 9907 raft_consensus.cc:686] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:43.716115 9907 consensus_queue.cc:237] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:43.717684 9882 consensus_queue.cc:1035] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.724782 9926 consensus_queue.cc:1035] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.727509 9917 consensus_queue.cc:1035] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.728516 9580 catalog_manager.cc:5484] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 1 to 2, leader changed from 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:43.752240 9899 consensus_queue.cc:1035] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.759487 9908 consensus_queue.cc:1035] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:04:43.955843 9908 raft_consensus.cc:979] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:43.956302 9908 raft_consensus.cc:1067] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:43.957913 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44186
I20240507 08:04:43.958508 9684 raft_consensus.cc:482] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:43.959229 9684 raft_consensus.cc:3037] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:43.964219 9684 raft_consensus.cc:504] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.965894 9684 leader_election.cc:288] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:43.966650 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:43.966971 9834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2152d4c8e1a948f4bf423fb3cac7fc3c" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:43.967191 9758 raft_consensus.cc:3037] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:43.967495 9834 raft_consensus.cc:3032] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:43.967744 9834 raft_consensus.cc:729] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:43.968318 9834 consensus_queue.cc:260] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.969201 9834 raft_consensus.cc:3037] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:43.971899 9758 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:43.972913 9650 leader_election.cc:302] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:43.973716 9907 raft_consensus.cc:2781] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:43.973635 9834 raft_consensus.cc:2454] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:43.974239 9907 raft_consensus.cc:686] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:43.975116 9907 consensus_queue.cc:237] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:43.982067 9582 catalog_manager.cc:5484] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 1 to 2, leader changed from 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:44.212862 9758 raft_consensus.cc:1261] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:44.214025 9878 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:44.221192 9834 raft_consensus.cc:1261] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:44.222852 9878 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:44.374500 9834 raft_consensus.cc:1261] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:44.375695 9878 consensus_queue.cc:1035] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:44.383774 9758 raft_consensus.cc:1261] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:44.385217 9878 consensus_queue.cc:1035] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:46.528066 9046 auto_leader_rebalancer-test.cc:331] The leader distribution is
I20240507 08:04:46.528749 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:46.550819 9684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
mode: GRACEFUL
new_leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:44210
I20240507 08:04:46.551358 9684 raft_consensus.cc:595] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Received request to transfer leadership to TS 16d2a767951f4624a42a1ffb7ed2a342
I20240507 08:04:46.553409 9684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
mode: GRACEFUL
new_leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:44210
I20240507 08:04:46.553886 9684 raft_consensus.cc:595] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Received request to transfer leadership to TS 16d2a767951f4624a42a1ffb7ed2a342
I20240507 08:04:46.554745 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240507 08:04:46.555155 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:46.680070 9971 raft_consensus.cc:979] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: : Instructing follower 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:04:46.680578 9878 raft_consensus.cc:1067] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Signalling peer 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:04:46.682050 9833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:58318
I20240507 08:04:46.682636 9833 raft_consensus.cc:482] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:46.682955 9833 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:46.688313 9833 raft_consensus.cc:504] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:46.689730 9833 leader_election.cc:288] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 2 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:46.690541 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:46.690743 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:46.691165 9684 raft_consensus.cc:3032] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:46.691274 9758 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:46.691449 9684 raft_consensus.cc:729] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:46.692152 9684 consensus_queue.cc:260] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:46.693010 9684 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:46.695870 9758 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 2.
I20240507 08:04:46.696835 9799 leader_election.cc:302] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:04:46.697482 9905 raft_consensus.cc:2781] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:46.697445 9684 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 2.
I20240507 08:04:46.698715 9905 raft_consensus.cc:686] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:46.699474 9905 consensus_queue.cc:237] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:46.706076 9582 catalog_manager.cc:5484] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 1 to 2, leader changed from e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129) to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 2 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:46.745569 9971 raft_consensus.cc:979] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: : Instructing follower 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:04:46.746031 9971 raft_consensus.cc:1067] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Signalling peer 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:04:46.747381 9833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:58318
I20240507 08:04:46.747838 9833 raft_consensus.cc:482] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:46.748111 9833 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:46.752414 9833 raft_consensus.cc:504] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:46.753980 9833 leader_election.cc:288] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 3 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:46.755023 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:46.755033 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:46.755554 9758 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:46.755616 9684 raft_consensus.cc:3032] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:46.755918 9684 raft_consensus.cc:729] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:46.756518 9684 consensus_queue.cc:260] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:46.757400 9684 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:46.760056 9758 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 3.
I20240507 08:04:46.761003 9799 leader_election.cc:302] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:04:46.761813 9905 raft_consensus.cc:2781] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:46.762192 9905 raft_consensus.cc:686] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:46.762235 9684 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 3.
I20240507 08:04:46.763015 9905 consensus_queue.cc:237] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:46.769884 9582 catalog_manager.cc:5484] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 2 to 3, leader changed from e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129) to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:47.171536 9684 raft_consensus.cc:1261] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:47.172681 9905 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:47.183053 9758 raft_consensus.cc:1261] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:47.184681 9905 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:47.273950 9684 raft_consensus.cc:1261] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:47.275323 9905 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:47.284081 9758 raft_consensus.cc:1261] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:47.285673 9905 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:47.556020 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:47.561270 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:47.561712 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:48.562547 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:48.568068 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:48.568506 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:49.569334 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:49.575543 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:49.575928 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:50.576633 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:50.582198 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:50.582670 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:51.583663 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:51.590463 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:51.590972 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:52.591849 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:52.597657 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:52.598088 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:53.598944 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:53.603989 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:53.604426 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:54.605314 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:54.610419 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:54.610788 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:55.611495 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:55.616410 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:04:55.616781 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:56.640508 9046 auto_leader_rebalancer-test.cc:350] The leader distribution is
I20240507 08:04:56.645057 9833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:58330
I20240507 08:04:56.645737 9833 raft_consensus.cc:595] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:56.649889 9833 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:58330
I20240507 08:04:56.650310 9833 raft_consensus.cc:595] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:56.676468 9758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4"
dest_uuid: "889812a87b2846a4979b1496ba962952"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:57562
I20240507 08:04:56.676951 9758 raft_consensus.cc:595] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:56.681326 9758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457"
dest_uuid: "889812a87b2846a4979b1496ba962952"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:57562
I20240507 08:04:56.681839 9758 raft_consensus.cc:595] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:56.686070 9758 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6"
dest_uuid: "889812a87b2846a4979b1496ba962952"
mode: GRACEFUL
new_leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:57562
I20240507 08:04:56.686501 9758 raft_consensus.cc:595] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Received request to transfer leadership to TS e9f8f88d16d04ce2b806a61e5c2e5d64
I20240507 08:04:56.776692 9992 raft_consensus.cc:979] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:56.777061 9992 raft_consensus.cc:1067] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:56.778443 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44200
I20240507 08:04:56.778937 9684 raft_consensus.cc:482] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:56.779311 9684 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:56.784729 9684 raft_consensus.cc:504] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:56.786357 9684 leader_election.cc:288] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:56.787348 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:56.787214 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:56.787894 9833 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:56.787958 9758 raft_consensus.cc:3032] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:56.788281 9758 raft_consensus.cc:729] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:56.788846 9758 consensus_queue.cc:260] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:56.789827 9758 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:56.792712 9833 raft_consensus.cc:2454] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:56.793816 9649 leader_election.cc:302] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:56.794572 9758 raft_consensus.cc:2454] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:56.794595 9999 raft_consensus.cc:2781] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:56.795722 9999 raft_consensus.cc:686] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:56.796620 9999 consensus_queue.cc:237] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:56.803517 9582 catalog_manager.cc:5484] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 1 to 2, leader changed from 889812a87b2846a4979b1496ba962952 (127.8.213.130) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:56.846675 10010 raft_consensus.cc:979] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:56.847188 9996 raft_consensus.cc:1067] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:56.848428 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44186
I20240507 08:04:56.848852 9684 raft_consensus.cc:482] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:56.849105 9684 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:56.853274 9684 raft_consensus.cc:504] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:56.854549 9684 leader_election.cc:288] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 4 election: Requested vote from peers 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:56.855419 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:56.855463 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5529e859954b9e822f87556c72c5df" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:56.855942 9833 raft_consensus.cc:3032] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:04:56.855962 9758 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:56.856298 9833 raft_consensus.cc:729] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:56.856946 9833 consensus_queue.cc:260] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:56.857852 9833 raft_consensus.cc:3037] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:56.860114 9758 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 4.
I20240507 08:04:56.861065 9650 leader_election.cc:302] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:56.861831 9999 raft_consensus.cc:2781] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:04:56.862187 9999 raft_consensus.cc:686] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:56.862215 9833 raft_consensus.cc:2454] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 4.
I20240507 08:04:56.863145 9999 consensus_queue.cc:237] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } }
I20240507 08:04:56.869258 9582 catalog_manager.cc:5484] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 3 to 4, leader changed from 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 4 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:56.887319 10010 raft_consensus.cc:979] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:56.887737 9996 raft_consensus.cc:1067] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:56.888952 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44186
I20240507 08:04:56.889359 9684 raft_consensus.cc:482] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:56.889652 9684 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:56.893867 9684 raft_consensus.cc:504] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:56.895368 9684 leader_election.cc:288] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 3 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:56.896131 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:56.896345 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:56.896695 9758 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:56.896797 9833 raft_consensus.cc:3032] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:56.897107 9833 raft_consensus.cc:729] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:56.897838 9833 consensus_queue.cc:260] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:56.898958 9833 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:56.902734 9758 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 3.
I20240507 08:04:56.903713 9650 leader_election.cc:302] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889812a87b2846a4979b1496ba962952, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:56.904372 9999 raft_consensus.cc:2781] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:56.904500 9833 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 3.
I20240507 08:04:56.904850 9999 raft_consensus.cc:686] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:56.905798 9999 consensus_queue.cc:237] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:56.911801 9582 catalog_manager.cc:5484] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 2 to 3, leader changed from 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 3 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:57.020411 10003 raft_consensus.cc:979] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:57.020846 9992 raft_consensus.cc:1067] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:57.022131 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44200
I20240507 08:04:57.022516 9684 raft_consensus.cc:482] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:57.022745 9684 raft_consensus.cc:3037] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:57.026647 9684 raft_consensus.cc:504] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:57.027859 9684 leader_election.cc:288] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:57.028718 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:57.028898 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a02c28220e04250a8f9c673bef04ae6" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:57.029143 9758 raft_consensus.cc:3032] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:57.029390 9758 raft_consensus.cc:729] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:57.029408 9833 raft_consensus.cc:3037] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:57.030107 9758 consensus_queue.cc:260] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:57.030975 9758 raft_consensus.cc:3037] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:57.033629 9833 raft_consensus.cc:2454] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:57.034464 9649 leader_election.cc:302] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:57.035107 9999 raft_consensus.cc:2781] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:57.035207 9758 raft_consensus.cc:2454] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:57.035485 9999 raft_consensus.cc:686] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:57.036410 9999 consensus_queue.cc:237] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:57.043048 9582 catalog_manager.cc:5484] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 1 to 2, leader changed from 889812a87b2846a4979b1496ba962952 (127.8.213.130) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:57.187613 10003 raft_consensus.cc:979] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: : Instructing follower e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:57.188036 9992 raft_consensus.cc:1067] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Signalling peer e9f8f88d16d04ce2b806a61e5c2e5d64 to start an election
I20240507 08:04:57.189450 9684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
from {username='slave'} at 127.0.0.1:44200
I20240507 08:04:57.189867 9758 raft_consensus.cc:1261] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:57.189900 9684 raft_consensus.cc:482] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:57.190320 9684 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:57.190994 9999 consensus_queue.cc:1035] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:57.195775 9684 raft_consensus.cc:504] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:57.197249 9684 leader_election.cc:288] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Requested vote from peers 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:57.198035 9758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:57.198541 9758 raft_consensus.cc:3032] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:04:57.198436 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:57.198882 9758 raft_consensus.cc:729] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:57.199007 9833 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:57.199620 9758 consensus_queue.cc:260] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:57.200846 9758 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:04:57.205608 9833 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:57.206633 9649 leader_election.cc:302] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, e9f8f88d16d04ce2b806a61e5c2e5d64; no voters:
I20240507 08:04:57.207100 9758 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f8f88d16d04ce2b806a61e5c2e5d64 in term 2.
I20240507 08:04:57.207664 10004 raft_consensus.cc:2781] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:04:57.208120 10004 raft_consensus.cc:686] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Leader. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:57.209044 10004 consensus_queue.cc:237] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:57.213858 9833 raft_consensus.cc:1261] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:57.221060 10018 consensus_queue.cc:1035] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:04:57.222748 9582 catalog_manager.cc:5484] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 reported cstate change: term changed from 1 to 2, leader changed from 889812a87b2846a4979b1496ba962952 (127.8.213.130) to e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129). New cstate: current_term: 2 leader_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:04:57.332075 9760 raft_consensus.cc:1261] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:04:57.333295 10019 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:04:57.342382 9833 raft_consensus.cc:1261] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:04:57.344467 10015 consensus_queue.cc:1035] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:04:57.450551 9833 raft_consensus.cc:1261] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:57.451725 10015 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:57.460585 9760 raft_consensus.cc:1261] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:04:57.461427 9758 raft_consensus.cc:1261] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:57.462478 10004 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:04:57.463239 10015 consensus_queue.cc:1035] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:57.469815 9833 raft_consensus.cc:1261] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:57.471601 10015 consensus_queue.cc:1035] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:57.590308 9834 raft_consensus.cc:1261] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:57.591488 10015 consensus_queue.cc:1035] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:57.599857 9760 raft_consensus.cc:1261] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Refusing update from remote peer e9f8f88d16d04ce2b806a61e5c2e5d64: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:04:57.601428 10015 consensus_queue.cc:1035] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:04:59.716537 9046 auto_leader_rebalancer-test.cc:365] The leader distribution is
I20240507 08:04:59.717249 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:04:59.724195 9684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
mode: GRACEFUL
new_leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:44210
I20240507 08:04:59.724707 9684 raft_consensus.cc:595] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 LEADER]: Received request to transfer leadership to TS 16d2a767951f4624a42a1ffb7ed2a342
I20240507 08:04:59.726920 9684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
mode: GRACEFUL
new_leader_uuid: "889812a87b2846a4979b1496ba962952"
from {username='slave'} at 127.0.0.1:44210
I20240507 08:04:59.727373 9684 raft_consensus.cc:595] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Received request to transfer leadership to TS 889812a87b2846a4979b1496ba962952
I20240507 08:04:59.729391 9684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
mode: GRACEFUL
new_leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:44210
I20240507 08:04:59.729835 9684 raft_consensus.cc:595] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Received request to transfer leadership to TS 16d2a767951f4624a42a1ffb7ed2a342
I20240507 08:04:59.731693 9684 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457"
dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
mode: GRACEFUL
new_leader_uuid: "889812a87b2846a4979b1496ba962952"
from {username='slave'} at 127.0.0.1:44210
I20240507 08:04:59.732106 9684 raft_consensus.cc:595] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Received request to transfer leadership to TS 889812a87b2846a4979b1496ba962952
I20240507 08:04:59.732941 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240507 08:04:59.733361 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:04:59.785486 10032 raft_consensus.cc:979] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: : Instructing follower 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:04:59.785954 10018 raft_consensus.cc:1067] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 LEADER]: Signalling peer 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:04:59.787159 9833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:58318
I20240507 08:04:59.787541 9833 raft_consensus.cc:482] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:59.787753 9833 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:59.792150 9833 raft_consensus.cc:504] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:59.793813 9833 leader_election.cc:288] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 4 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:04:59.794777 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:04:59.794656 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08be1fa1f2c6414bababd3c25c338d4e" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:59.795298 9760 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:59.795357 9684 raft_consensus.cc:3032] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:04:59.795661 9684 raft_consensus.cc:729] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:59.796232 9684 consensus_queue.cc:260] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:59.797091 9684 raft_consensus.cc:3037] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:04:59.801131 9760 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 4.
I20240507 08:04:59.802402 9799 leader_election.cc:302] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:04:59.803089 9996 raft_consensus.cc:2781] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:04:59.803283 9684 raft_consensus.cc:2454] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 4.
I20240507 08:04:59.803603 10032 raft_consensus.cc:979] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: : Instructing follower 889812a87b2846a4979b1496ba962952 to start an election
I20240507 08:04:59.804113 10033 raft_consensus.cc:1067] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Signalling peer 889812a87b2846a4979b1496ba962952 to start an election
I20240507 08:04:59.804435 9996 raft_consensus.cc:686] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:04:59.805269 9996 consensus_queue.cc:237] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:59.805737 9760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457"
dest_uuid: "889812a87b2846a4979b1496ba962952"
from {username='slave'} at 127.0.0.1:37764
I20240507 08:04:59.806138 9760 raft_consensus.cc:482] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:04:59.806356 9760 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:59.812470 9760 raft_consensus.cc:504] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:59.814539 9760 leader_election.cc:288] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 3 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:04:59.815306 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:04:59.815491 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9d1cca6e264060bbf1182f137d9457" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:04:59.814644 9583 catalog_manager.cc:5484] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 3 to 4, leader changed from e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129) to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 4 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } }
I20240507 08:04:59.815834 9684 raft_consensus.cc:3032] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:04:59.815969 9833 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:59.816186 9684 raft_consensus.cc:729] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:04:59.816836 9684 consensus_queue.cc:260] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:59.817939 9684 raft_consensus.cc:3037] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:04:59.821269 9833 raft_consensus.cc:2454] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 3.
I20240507 08:04:59.822237 9684 raft_consensus.cc:2454] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 3.
I20240507 08:04:59.822556 9724 leader_election.cc:302] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:04:59.823320 10031 raft_consensus.cc:2781] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:04:59.824810 10031 raft_consensus.cc:686] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 3 LEADER]: Becoming Leader. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:04:59.825544 10031 consensus_queue.cc:237] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:04:59.834182 9583 catalog_manager.cc:5484] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 reported cstate change: term changed from 2 to 3, leader changed from e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129) to 889812a87b2846a4979b1496ba962952 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "889812a87b2846a4979b1496ba962952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:00.128538 10018 raft_consensus.cc:979] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: : Instructing follower 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:05:00.128976 10033 raft_consensus.cc:1067] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Signalling peer 16d2a767951f4624a42a1ffb7ed2a342 to start an election
I20240507 08:05:00.130476 9833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4"
dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
from {username='slave'} at 127.0.0.1:58318
I20240507 08:05:00.130898 9833 raft_consensus.cc:482] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:05:00.131166 9833 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:00.135558 9833 raft_consensus.cc:504] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:05:00.136907 9833 leader_election.cc:288] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 3 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 889812a87b2846a4979b1496ba962952 (127.8.213.130:42723)
I20240507 08:05:00.137720 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:05:00.137930 9760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e5590e59cc48168ab470d065e905c4" candidate_uuid: "16d2a767951f4624a42a1ffb7ed2a342" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "889812a87b2846a4979b1496ba962952"
I20240507 08:05:00.138190 9684 raft_consensus.cc:3032] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:00.138376 9760 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:00.138479 9684 raft_consensus.cc:729] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:05:00.139075 9684 consensus_queue.cc:260] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:05:00.139858 9684 raft_consensus.cc:3037] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:00.142455 9760 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 3.
I20240507 08:05:00.143253 9799 leader_election.cc:302] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:05:00.143883 9996 raft_consensus.cc:2781] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:00.144011 9684 raft_consensus.cc:2454] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16d2a767951f4624a42a1ffb7ed2a342 in term 3.
I20240507 08:05:00.144407 9996 raft_consensus.cc:686] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Becoming Leader. State: Replica: 16d2a767951f4624a42a1ffb7ed2a342, State: Running, Role: LEADER
I20240507 08:05:00.145367 9996 consensus_queue.cc:237] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:05:00.151348 9583 catalog_manager.cc:5484] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 reported cstate change: term changed from 2 to 3, leader changed from e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129) to 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "16d2a767951f4624a42a1ffb7ed2a342" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:00.218936 9760 raft_consensus.cc:1261] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:05:00.219964 9996 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:00.231376 9684 raft_consensus.cc:1261] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:05:00.232770 9996 consensus_queue.cc:1035] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:00.244781 10033 raft_consensus.cc:979] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: : Instructing follower 889812a87b2846a4979b1496ba962952 to start an election
I20240507 08:05:00.245236 10018 raft_consensus.cc:1067] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Signalling peer 889812a87b2846a4979b1496ba962952 to start an election
I20240507 08:05:00.246553 9760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab"
dest_uuid: "889812a87b2846a4979b1496ba962952"
from {username='slave'} at 127.0.0.1:37764
I20240507 08:05:00.247033 9760 raft_consensus.cc:482] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:05:00.247367 9760 raft_consensus.cc:3037] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:00.251652 9760 raft_consensus.cc:504] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:05:00.252892 9760 leader_election.cc:288] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 2 election: Requested vote from peers e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337), 16d2a767951f4624a42a1ffb7ed2a342 (127.8.213.131:36017)
I20240507 08:05:00.253651 9684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64"
I20240507 08:05:00.253780 9833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7146f803bde54a5889435a67cfe34cab" candidate_uuid: "889812a87b2846a4979b1496ba962952" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "16d2a767951f4624a42a1ffb7ed2a342"
I20240507 08:05:00.254204 9684 raft_consensus.cc:3032] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:00.254221 9833 raft_consensus.cc:3037] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:00.254549 9684 raft_consensus.cc:729] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e9f8f88d16d04ce2b806a61e5c2e5d64, State: Running, Role: LEADER
I20240507 08:05:00.255123 9684 consensus_queue.cc:260] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:05:00.255935 9684 raft_consensus.cc:3037] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:00.258284 9833 raft_consensus.cc:2454] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 2.
I20240507 08:05:00.259171 9724 leader_election.cc:302] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16d2a767951f4624a42a1ffb7ed2a342, 889812a87b2846a4979b1496ba962952; no voters:
I20240507 08:05:00.259896 10031 raft_consensus.cc:2781] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:00.260150 9684 raft_consensus.cc:2454] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889812a87b2846a4979b1496ba962952 in term 2.
I20240507 08:05:00.260236 10031 raft_consensus.cc:686] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 2 LEADER]: Becoming Leader. State: Replica: 889812a87b2846a4979b1496ba962952, State: Running, Role: LEADER
I20240507 08:05:00.261029 10031 consensus_queue.cc:237] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } }
I20240507 08:05:00.267634 9580 catalog_manager.cc:5484] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 reported cstate change: term changed from 1 to 2, leader changed from e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129) to 889812a87b2846a4979b1496ba962952 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "889812a87b2846a4979b1496ba962952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:00.300922 9833 raft_consensus.cc:1261] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Refusing update from remote peer 889812a87b2846a4979b1496ba962952: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:00.301991 10031 consensus_queue.cc:1035] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:00.309321 9684 raft_consensus.cc:1261] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Refusing update from remote peer 889812a87b2846a4979b1496ba962952: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:00.310855 10050 consensus_queue.cc:1035] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:00.674109 9684 raft_consensus.cc:1261] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Refusing update from remote peer 889812a87b2846a4979b1496ba962952: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:00.675428 10050 consensus_queue.cc:1035] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:00.683578 9833 raft_consensus.cc:1261] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Refusing update from remote peer 889812a87b2846a4979b1496ba962952: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:00.685293 10031 consensus_queue.cc:1035] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16d2a767951f4624a42a1ffb7ed2a342" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 36017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:00.724392 9760 raft_consensus.cc:1261] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:00.725442 9996 consensus_queue.cc:1035] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "889812a87b2846a4979b1496ba962952" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:00.732956 9684 raft_consensus.cc:1261] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Refusing update from remote peer 16d2a767951f4624a42a1ffb7ed2a342: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:00.734237 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:00.734362 10055 consensus_queue.cc:1035] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f8f88d16d04ce2b806a61e5c2e5d64" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 40337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:00.744179 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:00.744758 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:01.745900 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:01.754088 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:01.754529 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:02.755420 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:02.761751 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:02.762176 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:03.763090 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:03.768764 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:03.769208 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:04.769948 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:04.775126 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:04.775535 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:05.776239 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:05.782625 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:05.783028 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:06.783726 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:06.788944 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:06.789374 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:07.790166 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:07.795284 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:07.795668 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:08.796651 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:08.802269 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240507 08:05:08.802673 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:09.825989 9046 auto_leader_rebalancer-test.cc:376] The leader distribution is
I20240507 08:05:09.826483 9046 tablet_server.cc:178] TabletServer@127.8.213.129:0 shutting down...
I20240507 08:05:09.855707 9046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:09.856599 9046 tablet_replica.cc:330] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.857275 9046 raft_consensus.cc:2229] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.858222 9046 raft_consensus.cc:2258] T 0a02c28220e04250a8f9c673bef04ae6 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.860143 9046 tablet_replica.cc:330] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.860647 9046 raft_consensus.cc:2229] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.861063 9046 raft_consensus.cc:2258] T 7146f803bde54a5889435a67cfe34cab P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.862700 9046 tablet_replica.cc:330] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.863204 9046 raft_consensus.cc:2229] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.863567 9046 raft_consensus.cc:2258] T eb9d1cca6e264060bbf1182f137d9457 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.865095 9046 tablet_replica.cc:330] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.865525 9046 raft_consensus.cc:2229] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.865936 9046 raft_consensus.cc:2258] T bdcca60a5c704607b61bce4d80e4d4b1 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.867383 9046 tablet_replica.cc:330] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.867864 9046 raft_consensus.cc:2229] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.868244 9046 raft_consensus.cc:2258] T 08be1fa1f2c6414bababd3c25c338d4e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.869836 9046 tablet_replica.cc:330] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.870293 9046 raft_consensus.cc:2229] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.870672 9046 raft_consensus.cc:2258] T e9e5590e59cc48168ab470d065e905c4 P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.872159 9046 tablet_replica.cc:330] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.872582 9046 raft_consensus.cc:2229] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.873281 9046 raft_consensus.cc:2258] T 2152d4c8e1a948f4bf423fb3cac7fc3c P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.875000 9046 tablet_replica.cc:330] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.875463 9046 raft_consensus.cc:2229] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.875797 9046 raft_consensus.cc:2258] T bfd011611a764aa880282abfd309fccd P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.877450 9046 tablet_replica.cc:330] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.878104 9046 raft_consensus.cc:2229] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.878978 9046 raft_consensus.cc:2258] T ee5529e859954b9e822f87556c72c5df P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240507 08:05:09.881136 9726 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
I20240507 08:05:09.881422 9046 tablet_replica.cc:330] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64: stopping tablet replica
I20240507 08:05:09.882213 9046 raft_consensus.cc:2229] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.883128 9046 raft_consensus.cc:2258] T c3f1b3e789e14494b2d37fbc8409184e P e9f8f88d16d04ce2b806a61e5c2e5d64 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:05:09.885054 9726 consensus_peers.cc:485] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 -> Peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Couldn't send request to peer e9f8f88d16d04ce2b806a61e5c2e5d64. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:09.913800 9046 tablet_server.cc:194] TabletServer@127.8.213.129:0 shutdown complete.
W20240507 08:05:09.920573 9726 consensus_peers.cc:485] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 -> Peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Couldn't send request to peer e9f8f88d16d04ce2b806a61e5c2e5d64. Status: Network error: Client connection negotiation failed: client connection to 127.8.213.129:40337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:09.931726 9046 tablet_server.cc:178] TabletServer@127.8.213.130:0 shutting down...
I20240507 08:05:09.956032 9046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:09.956656 9046 tablet_replica.cc:330] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.957378 9046 raft_consensus.cc:2229] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.957949 9046 raft_consensus.cc:2258] T e9e5590e59cc48168ab470d065e905c4 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.959648 9046 tablet_replica.cc:330] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.960101 9046 raft_consensus.cc:2229] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 2 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.960803 9046 raft_consensus.cc:2258] T 7146f803bde54a5889435a67cfe34cab P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.962489 9046 tablet_replica.cc:330] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.962949 9046 raft_consensus.cc:2229] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.963310 9046 raft_consensus.cc:2258] T 0a02c28220e04250a8f9c673bef04ae6 P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.964927 9046 tablet_replica.cc:330] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.965476 9046 raft_consensus.cc:2229] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.965926 9046 raft_consensus.cc:2258] T bdcca60a5c704607b61bce4d80e4d4b1 P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.967581 9046 tablet_replica.cc:330] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.968050 9046 raft_consensus.cc:2229] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.968732 9046 raft_consensus.cc:2258] T eb9d1cca6e264060bbf1182f137d9457 P 889812a87b2846a4979b1496ba962952 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.970301 9046 tablet_replica.cc:330] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.970746 9046 raft_consensus.cc:2229] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.971130 9046 raft_consensus.cc:2258] T 08be1fa1f2c6414bababd3c25c338d4e P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.972628 9046 tablet_replica.cc:330] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.973071 9046 raft_consensus.cc:2229] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.973461 9046 raft_consensus.cc:2258] T c3f1b3e789e14494b2d37fbc8409184e P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.975104 9046 tablet_replica.cc:330] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.975539 9046 raft_consensus.cc:2229] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.975994 9046 raft_consensus.cc:2258] T ee5529e859954b9e822f87556c72c5df P 889812a87b2846a4979b1496ba962952 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.977510 9046 tablet_replica.cc:330] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.977945 9046 raft_consensus.cc:2229] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:09.978657 9046 raft_consensus.cc:2258] T bfd011611a764aa880282abfd309fccd P 889812a87b2846a4979b1496ba962952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:09.980257 9046 tablet_replica.cc:330] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952: stopping tablet replica
I20240507 08:05:09.980669 9046 raft_consensus.cc:2229] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:09.981053 9046 raft_consensus.cc:2258] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 889812a87b2846a4979b1496ba962952 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.000754 9046 tablet_server.cc:194] TabletServer@127.8.213.130:0 shutdown complete.
W20240507 08:05:10.001690 9800 consensus_peers.cc:485] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 -> Peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Couldn't send request to peer e9f8f88d16d04ce2b806a61e5c2e5d64. Status: Network error: Client connection negotiation failed: client connection to 127.8.213.129:40337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:10.019104 9046 tablet_server.cc:178] TabletServer@127.8.213.131:0 shutting down...
W20240507 08:05:10.024016 9800 consensus_peers.cc:485] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 -> Peer e9f8f88d16d04ce2b806a61e5c2e5d64 (127.8.213.129:40337): Couldn't send request to peer e9f8f88d16d04ce2b806a61e5c2e5d64. Status: Network error: Client connection negotiation failed: client connection to 127.8.213.129:40337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:10.046758 9046 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240507 08:05:10.047426 9046 tablet_replica.cc:330] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.048017 9046 raft_consensus.cc:2229] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 LEADER]: Raft consensus shutting down.
I20240507 08:05:10.048826 9046 raft_consensus.cc:2258] T e9e5590e59cc48168ab470d065e905c4 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.050639 9046 tablet_replica.cc:330] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.051131 9046 raft_consensus.cc:2229] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 LEADER]: Raft consensus shutting down.
I20240507 08:05:10.051849 9046 raft_consensus.cc:2258] T 08be1fa1f2c6414bababd3c25c338d4e P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.053648 9046 tablet_replica.cc:330] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.054101 9046 raft_consensus.cc:2229] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.054484 9046 raft_consensus.cc:2258] T 0a02c28220e04250a8f9c673bef04ae6 P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.056100 9046 tablet_replica.cc:330] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.056603 9046 raft_consensus.cc:2229] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.056985 9046 raft_consensus.cc:2258] T c3f1b3e789e14494b2d37fbc8409184e P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.058543 9046 tablet_replica.cc:330] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.058970 9046 raft_consensus.cc:2229] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.059362 9046 raft_consensus.cc:2258] T ee5529e859954b9e822f87556c72c5df P 16d2a767951f4624a42a1ffb7ed2a342 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.060863 9046 tablet_replica.cc:330] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.061295 9046 raft_consensus.cc:2229] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.061678 9046 raft_consensus.cc:2258] T eb9d1cca6e264060bbf1182f137d9457 P 16d2a767951f4624a42a1ffb7ed2a342 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.063212 9046 tablet_replica.cc:330] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.063658 9046 raft_consensus.cc:2229] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:10.064343 9046 raft_consensus.cc:2258] T bdcca60a5c704607b61bce4d80e4d4b1 P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.066084 9046 tablet_replica.cc:330] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.066504 9046 raft_consensus.cc:2229] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.066866 9046 raft_consensus.cc:2258] T bfd011611a764aa880282abfd309fccd P 16d2a767951f4624a42a1ffb7ed2a342 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.068318 9046 tablet_replica.cc:330] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.068740 9046 raft_consensus.cc:2229] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.069113 9046 raft_consensus.cc:2258] T 2152d4c8e1a948f4bf423fb3cac7fc3c P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.070581 9046 tablet_replica.cc:330] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342: stopping tablet replica
I20240507 08:05:10.070981 9046 raft_consensus.cc:2229] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:10.071365 9046 raft_consensus.cc:2258] T 7146f803bde54a5889435a67cfe34cab P 16d2a767951f4624a42a1ffb7ed2a342 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.099222 9046 tablet_server.cc:194] TabletServer@127.8.213.131:0 shutdown complete.
I20240507 08:05:10.116201 9046 master.cc:537] Master@127.8.213.190:44427 shutting down...
I20240507 08:05:10.141726 9046 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:05:10.142238 9046 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:10.142556 9046 tablet_replica.cc:330] T 00000000000000000000000000000000 P e86bc57feda242dda37183adf4ad14f7: stopping tablet replica
I20240507 08:05:10.160974 9046 master.cc:552] Master@127.8.213.190:44427 shutdown complete.
16d2a767951f4624a42a1ffb7ed2a342 1
889812a87b2846a4979b1496ba962952 4
e9f8f88d16d04ce2b806a61e5c2e5d64 5
16d2a767951f4624a42a1ffb7ed2a342 3
889812a87b2846a4979b1496ba962952 4
e9f8f88d16d04ce2b806a61e5c2e5d64 3
16d2a767951f4624a42a1ffb7ed2a342 1
889812a87b2846a4979b1496ba962952 1
e9f8f88d16d04ce2b806a61e5c2e5d64 8
16d2a767951f4624a42a1ffb7ed2a342 3
889812a87b2846a4979b1496ba962952 3
e9f8f88d16d04ce2b806a61e5c2e5d64 4
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28550 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20240507 08:05:10.209909 9046 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.8.213.190:37115
I20240507 08:05:10.211032 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:10.217010 10065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.217689 10066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.218400 10068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:10.218931 9046 server_base.cc:1034] running on GCE node
I20240507 08:05:10.219805 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:10.219985 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:10.220121 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069110220105 us; error 0 us; skew 500 ppm
I20240507 08:05:10.220634 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:10.223187 9046 webserver.cc:460] Webserver started at http://127.8.213.190:43503/ using document root <none> and password file <none>
I20240507 08:05:10.223659 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:10.223840 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:10.224105 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:10.225255 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/master-0-root/instance:
uuid: "9ee8354230fc476f93816bfbab26bf0a"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.229557 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240507 08:05:10.232748 10073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.233422 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240507 08:05:10.233705 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/master-0-root
uuid: "9ee8354230fc476f93816bfbab26bf0a"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.233975 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:10.274681 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:05:10.275908 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:10.311913 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.190:37115
I20240507 08:05:10.312008 10124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.190:37115 every 8 connection(s)
I20240507 08:05:10.315711 10125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.327620 10125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Bootstrap starting.
I20240507 08:05:10.331692 10125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:10.335737 10125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: No bootstrap required, opened a new log
I20240507 08:05:10.337713 10125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8354230fc476f93816bfbab26bf0a" member_type: VOTER }
I20240507 08:05:10.338202 10125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:10.338379 10125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee8354230fc476f93816bfbab26bf0a, State: Initialized, Role: FOLLOWER
I20240507 08:05:10.338852 10125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8354230fc476f93816bfbab26bf0a" member_type: VOTER }
I20240507 08:05:10.339272 10125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240507 08:05:10.339452 10125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240507 08:05:10.339666 10125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:10.343609 10125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8354230fc476f93816bfbab26bf0a" member_type: VOTER }
I20240507 08:05:10.344063 10125 leader_election.cc:302] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ee8354230fc476f93816bfbab26bf0a; no voters:
I20240507 08:05:10.345029 10125 leader_election.cc:288] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240507 08:05:10.345397 10128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:10.346685 10128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [term 1 LEADER]: Becoming Leader. State: Replica: 9ee8354230fc476f93816bfbab26bf0a, State: Running, Role: LEADER
I20240507 08:05:10.347359 10128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8354230fc476f93816bfbab26bf0a" member_type: VOTER }
I20240507 08:05:10.347836 10125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:05:10.351629 10129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ee8354230fc476f93816bfbab26bf0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8354230fc476f93816bfbab26bf0a" member_type: VOTER } }
I20240507 08:05:10.352149 10129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:10.353309 10130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ee8354230fc476f93816bfbab26bf0a. Latest consensus state: current_term: 1 leader_uuid: "9ee8354230fc476f93816bfbab26bf0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee8354230fc476f93816bfbab26bf0a" member_type: VOTER } }
I20240507 08:05:10.353910 10130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a [sys.catalog]: This master's current role is: LEADER
I20240507 08:05:10.360306 10136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:05:10.367488 10136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:05:10.368466 9046 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240507 08:05:10.375893 10136 catalog_manager.cc:1347] Generated new cluster ID: 530348387b464e41ad1727dc283a9476
I20240507 08:05:10.376137 10136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:05:10.388666 10136 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:05:10.389828 10136 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:05:10.403174 10136 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Generated new TSK 0
I20240507 08:05:10.403714 10136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:05:10.435683 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:10.442456 10146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.443490 10147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.444330 10149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:10.445027 9046 server_base.cc:1034] running on GCE node
I20240507 08:05:10.445837 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:10.446019 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:10.446158 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069110446144 us; error 0 us; skew 500 ppm
I20240507 08:05:10.446722 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:10.449082 9046 webserver.cc:460] Webserver started at http://127.8.213.129:42131/ using document root <none> and password file <none>
I20240507 08:05:10.449551 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:10.449762 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:10.450001 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:10.451049 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/instance:
uuid: "59382e6647c549d89d83c70233c0667e"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.455397 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240507 08:05:10.458444 10154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.459084 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240507 08:05:10.459388 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root
uuid: "59382e6647c549d89d83c70233c0667e"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.459690 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:10.472965 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:05:10.474180 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:10.475555 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:10.477737 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:10.477921 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.478147 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:10.478308 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.515463 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.129:32795
I20240507 08:05:10.515549 10216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.129:32795 every 8 connection(s)
I20240507 08:05:10.519820 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:10.527891 10221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.529824 10222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.530635 10224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:10.532974 10217 heartbeater.cc:347] Connected to a master server at 127.8.213.190:37115
I20240507 08:05:10.533293 10217 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:10.533756 9046 server_base.cc:1034] running on GCE node
I20240507 08:05:10.534049 10217 heartbeater.cc:511] Master 127.8.213.190:37115 requested a full tablet report, sending...
I20240507 08:05:10.535317 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:10.535594 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:10.535794 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069110535776 us; error 0 us; skew 500 ppm
I20240507 08:05:10.536214 10090 ts_manager.cc:194] Registered new tserver with Master: 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:05:10.536648 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:10.538656 10090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59936
I20240507 08:05:10.539465 9046 webserver.cc:460] Webserver started at http://127.8.213.130:41277/ using document root <none> and password file <none>
I20240507 08:05:10.540104 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:10.540319 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:10.540567 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:10.541747 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/instance:
uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.545944 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240507 08:05:10.549003 10229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.549733 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240507 08:05:10.549990 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root
uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.550251 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:10.558794 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:05:10.559916 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:10.561383 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:10.563489 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:10.563678 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.563895 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:10.564042 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.600878 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.130:41081
I20240507 08:05:10.600978 10291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.130:41081 every 8 connection(s)
I20240507 08:05:10.605425 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:10.612442 10295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:10.614261 10296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:10.616441 10292 heartbeater.cc:347] Connected to a master server at 127.8.213.190:37115
I20240507 08:05:10.616786 10292 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:10.616863 9046 server_base.cc:1034] running on GCE node
W20240507 08:05:10.617599 10298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:10.617699 10292 heartbeater.cc:511] Master 127.8.213.190:37115 requested a full tablet report, sending...
I20240507 08:05:10.618660 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:10.618865 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:10.619038 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069110619021 us; error 0 us; skew 500 ppm
I20240507 08:05:10.619593 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:10.619686 10090 ts_manager.cc:194] Registered new tserver with Master: bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:10.621284 10090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59948
I20240507 08:05:10.622200 9046 webserver.cc:460] Webserver started at http://127.8.213.131:44985/ using document root <none> and password file <none>
I20240507 08:05:10.622721 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:10.622903 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:10.623137 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:10.624212 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/instance:
uuid: "88a26dac37fa4243b8c2b3d47f403488"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.628448 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240507 08:05:10.631413 10303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.632097 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240507 08:05:10.632396 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root
uuid: "88a26dac37fa4243b8c2b3d47f403488"
format_stamp: "Formatted at 2024-05-07 08:05:10 on dist-test-slave-ls56"
I20240507 08:05:10.632719 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:10.647495 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:05:10.648665 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:10.650300 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:10.652580 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:10.652776 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.652983 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:10.653132 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:10.690773 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.131:37047
I20240507 08:05:10.690871 10365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.131:37047 every 8 connection(s)
I20240507 08:05:10.703413 10366 heartbeater.cc:347] Connected to a master server at 127.8.213.190:37115
I20240507 08:05:10.703806 10366 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:10.704495 10366 heartbeater.cc:511] Master 127.8.213.190:37115 requested a full tablet report, sending...
I20240507 08:05:10.706216 10090 ts_manager.cc:194] Registered new tserver with Master: 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:10.707111 9046 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01333047s
I20240507 08:05:10.707473 10090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59954
I20240507 08:05:10.709340 9046 test_util.cc:268] Using random seed: -1524896704
I20240507 08:05:10.730310 10090 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59970:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:05:10.736680 10090 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:05:10.930773 10331 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5fd8588b9240c38e1f3c75cf342156 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240507 08:05:10.932224 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5fd8588b9240c38e1f3c75cf342156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.932415 10330 tablet_service.cc:1467] Processing CreateTablet for tablet ffd4c65b5ad04f48916c3e359f7e45c9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240507 08:05:10.934183 10182 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5fd8588b9240c38e1f3c75cf342156 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240507 08:05:10.934757 10329 tablet_service.cc:1467] Processing CreateTablet for tablet 82c5eb4243284619ad55e8a724ff32de (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240507 08:05:10.935429 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5fd8588b9240c38e1f3c75cf342156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.937348 10178 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc98f346bff4d428582b1ad1dc0fc93 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240507 08:05:10.937490 10328 tablet_service.cc:1467] Processing CreateTablet for tablet 7b89492a3abf415da273cc9a5b1227bf (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240507 08:05:10.938704 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc98f346bff4d428582b1ad1dc0fc93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.938818 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd4c65b5ad04f48916c3e359f7e45c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.939858 10180 tablet_service.cc:1467] Processing CreateTablet for tablet 82c5eb4243284619ad55e8a724ff32de (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240507 08:05:10.941170 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c5eb4243284619ad55e8a724ff32de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.941933 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b89492a3abf415da273cc9a5b1227bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.946745 10179 tablet_service.cc:1467] Processing CreateTablet for tablet 7b89492a3abf415da273cc9a5b1227bf (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240507 08:05:10.947973 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c5eb4243284619ad55e8a724ff32de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.938730 10181 tablet_service.cc:1467] Processing CreateTablet for tablet ffd4c65b5ad04f48916c3e359f7e45c9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240507 08:05:10.947994 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b89492a3abf415da273cc9a5b1227bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.949631 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd4c65b5ad04f48916c3e359f7e45c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.951583 10327 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc98f346bff4d428582b1ad1dc0fc93 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240507 08:05:10.952839 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc98f346bff4d428582b1ad1dc0fc93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.975229 10177 tablet_service.cc:1467] Processing CreateTablet for tablet db7c60767d74458094c1d86a49e87980 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240507 08:05:10.976594 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c60767d74458094c1d86a49e87980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.980055 10326 tablet_service.cc:1467] Processing CreateTablet for tablet db7c60767d74458094c1d86a49e87980 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240507 08:05:10.980448 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 10eece2dd4a64e3aa99d91fadc43df89 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240507 08:05:10.981406 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c60767d74458094c1d86a49e87980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.982631 10175 tablet_service.cc:1467] Processing CreateTablet for tablet 67fb9faf0618484ab4dd4cfb90e69c28 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240507 08:05:10.982698 10325 tablet_service.cc:1467] Processing CreateTablet for tablet 10eece2dd4a64e3aa99d91fadc43df89 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240507 08:05:10.986011 10324 tablet_service.cc:1467] Processing CreateTablet for tablet 67fb9faf0618484ab4dd4cfb90e69c28 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240507 08:05:10.986939 10181 tablet_service.cc:1467] Processing CreateTablet for tablet beb5387c55024337b72bb8aa1ba09a7b (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240507 08:05:10.987425 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10eece2dd4a64e3aa99d91fadc43df89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.991201 10325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10eece2dd4a64e3aa99d91fadc43df89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:10.994086 10324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fb9faf0618484ab4dd4cfb90e69c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.001684 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 2e5fd8588b9240c38e1f3c75cf342156 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20240507 08:05:11.003044 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e5fd8588b9240c38e1f3c75cf342156. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.003654 10249 tablet_service.cc:1467] Processing CreateTablet for tablet beb5387c55024337b72bb8aa1ba09a7b (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240507 08:05:11.005177 10256 tablet_service.cc:1467] Processing CreateTablet for tablet ffd4c65b5ad04f48916c3e359f7e45c9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20240507 08:05:11.005661 10180 tablet_service.cc:1467] Processing CreateTablet for tablet 64632a3e07184039950f3ef7fe699cb3 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240507 08:05:11.007328 10255 tablet_service.cc:1467] Processing CreateTablet for tablet 82c5eb4243284619ad55e8a724ff32de (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20240507 08:05:11.007825 10248 tablet_service.cc:1467] Processing CreateTablet for tablet 64632a3e07184039950f3ef7fe699cb3 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240507 08:05:11.009500 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 7b89492a3abf415da273cc9a5b1227bf (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20240507 08:05:11.010177 10179 tablet_service.cc:1467] Processing CreateTablet for tablet 316d26400fa14c4b9e67f139ffeca8f1 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240507 08:05:11.010550 10327 tablet_service.cc:1467] Processing CreateTablet for tablet 316d26400fa14c4b9e67f139ffeca8f1 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240507 08:05:11.011736 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316d26400fa14c4b9e67f139ffeca8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.011639 10253 tablet_service.cc:1467] Processing CreateTablet for tablet 7dc98f346bff4d428582b1ad1dc0fc93 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20240507 08:05:10.994086 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fb9faf0618484ab4dd4cfb90e69c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.014057 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316d26400fa14c4b9e67f139ffeca8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.014554 10250 tablet_service.cc:1467] Processing CreateTablet for tablet 67fb9faf0618484ab4dd4cfb90e69c28 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20240507 08:05:11.015792 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67fb9faf0618484ab4dd4cfb90e69c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.017937 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64632a3e07184039950f3ef7fe699cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.017648 10251 tablet_service.cc:1467] Processing CreateTablet for tablet 10eece2dd4a64e3aa99d91fadc43df89 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20240507 08:05:11.018918 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10eece2dd4a64e3aa99d91fadc43df89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.019464 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64632a3e07184039950f3ef7fe699cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.021904 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c5eb4243284619ad55e8a724ff32de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.022722 10252 tablet_service.cc:1467] Processing CreateTablet for tablet db7c60767d74458094c1d86a49e87980 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20240507 08:05:11.024327 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd4c65b5ad04f48916c3e359f7e45c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.027962 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b89492a3abf415da273cc9a5b1227bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.031618 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c60767d74458094c1d86a49e87980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.034006 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dc98f346bff4d428582b1ad1dc0fc93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.043009 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5387c55024337b72bb8aa1ba09a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.054714 10329 tablet_service.cc:1467] Processing CreateTablet for tablet 64632a3e07184039950f3ef7fe699cb3 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20240507 08:05:11.056376 10323 tablet_service.cc:1467] Processing CreateTablet for tablet 19878aa675724839ac9bb59333dee2d9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240507 08:05:11.056633 10174 tablet_service.cc:1467] Processing CreateTablet for tablet 19878aa675724839ac9bb59333dee2d9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240507 08:05:11.057809 10323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19878aa675724839ac9bb59333dee2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.058369 10382 tablet_bootstrap.cc:492] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:11.058773 10173 tablet_service.cc:1467] Processing CreateTablet for tablet 3df41f82bc854e7c9a22f3f345b03f92 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240507 08:05:11.062389 10381 tablet_bootstrap.cc:492] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:11.064818 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64632a3e07184039950f3ef7fe699cb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.054581 10328 tablet_service.cc:1467] Processing CreateTablet for tablet beb5387c55024337b72bb8aa1ba09a7b (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20240507 08:05:11.073220 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 3df41f82bc854e7c9a22f3f345b03f92 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240507 08:05:11.074923 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5387c55024337b72bb8aa1ba09a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.075946 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df41f82bc854e7c9a22f3f345b03f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.080215 10381 tablet_bootstrap.cc:654] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.092664 10178 tablet_service.cc:1467] Processing CreateTablet for tablet ed7ce583c04c48ebb2e80d2e0e872215 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240507 08:05:11.092955 10331 tablet_service.cc:1467] Processing CreateTablet for tablet ed7ce583c04c48ebb2e80d2e0e872215 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240507 08:05:11.094403 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7ce583c04c48ebb2e80d2e0e872215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.109310 10330 tablet_service.cc:1467] Processing CreateTablet for tablet 9800706282e543be9f2ee3473da2b89f (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240507 08:05:11.111990 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9800706282e543be9f2ee3473da2b89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.118925 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 9800706282e543be9f2ee3473da2b89f (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240507 08:05:11.125023 10179 tablet_service.cc:1467] Processing CreateTablet for tablet 0c898839b7c84ce6ba0e135f778ad72d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240507 08:05:11.126753 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5387c55024337b72bb8aa1ba09a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.127398 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19878aa675724839ac9bb59333dee2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.128036 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9800706282e543be9f2ee3473da2b89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.128616 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7ce583c04c48ebb2e80d2e0e872215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.133184 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df41f82bc854e7c9a22f3f345b03f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.141881 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c898839b7c84ce6ba0e135f778ad72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.145206 10252 tablet_service.cc:1467] Processing CreateTablet for tablet 316d26400fa14c4b9e67f139ffeca8f1 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20240507 08:05:11.146493 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 316d26400fa14c4b9e67f139ffeca8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.154395 10382 tablet_bootstrap.cc:654] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.154817 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 0c898839b7c84ce6ba0e135f778ad72d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240507 08:05:11.158655 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c898839b7c84ce6ba0e135f778ad72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.163520 10249 tablet_service.cc:1467] Processing CreateTablet for tablet ed7ce583c04c48ebb2e80d2e0e872215 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20240507 08:05:11.165701 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 9800706282e543be9f2ee3473da2b89f (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20240507 08:05:11.167392 10255 tablet_service.cc:1467] Processing CreateTablet for tablet 0c898839b7c84ce6ba0e135f778ad72d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20240507 08:05:11.157272 10256 tablet_service.cc:1467] Processing CreateTablet for tablet 19878aa675724839ac9bb59333dee2d9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20240507 08:05:11.157029 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 3df41f82bc854e7c9a22f3f345b03f92 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20240507 08:05:11.174598 10248 tablet_service.cc:1467] Processing CreateTablet for tablet 50d914d033cf48c389262dd8b7c7fbce (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240507 08:05:11.179723 10182 tablet_service.cc:1467] Processing CreateTablet for tablet 50d914d033cf48c389262dd8b7c7fbce (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240507 08:05:11.181493 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d914d033cf48c389262dd8b7c7fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.188673 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 7a181f4ae4d342129ad262e5545debc9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240507 08:05:11.189965 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a181f4ae4d342129ad262e5545debc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.189867 10331 tablet_service.cc:1467] Processing CreateTablet for tablet 50d914d033cf48c389262dd8b7c7fbce (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20240507 08:05:11.191066 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d914d033cf48c389262dd8b7c7fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.195396 10381 tablet_bootstrap.cc:492] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:11.195888 10381 ts_tablet_manager.cc:1367] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.006s
I20240507 08:05:11.197149 10329 tablet_service.cc:1467] Processing CreateTablet for tablet 7a181f4ae4d342129ad262e5545debc9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240507 08:05:11.198397 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a181f4ae4d342129ad262e5545debc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.198585 10181 tablet_service.cc:1467] Processing CreateTablet for tablet 64f8f2cf89ed4b99822bc4389d7a842f (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240507 08:05:11.199788 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f8f2cf89ed4b99822bc4389d7a842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.200646 10323 tablet_service.cc:1467] Processing CreateTablet for tablet 64f8f2cf89ed4b99822bc4389d7a842f (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240507 08:05:11.201854 10323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f8f2cf89ed4b99822bc4389d7a842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.203935 10381 raft_consensus.cc:348] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:11.204689 10381 raft_consensus.cc:374] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:11.204972 10381 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:11.210253 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c898839b7c84ce6ba0e135f778ad72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.211556 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7ce583c04c48ebb2e80d2e0e872215. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.211212 10179 tablet_service.cc:1467] Processing CreateTablet for tablet a8fba2019b474ce4b80d6b6d7c81a2ff (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240507 08:05:11.212556 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fba2019b474ce4b80d6b6d7c81a2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.214982 10253 tablet_service.cc:1467] Processing CreateTablet for tablet 64f8f2cf89ed4b99822bc4389d7a842f (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20240507 08:05:11.216207 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64f8f2cf89ed4b99822bc4389d7a842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.219959 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d914d033cf48c389262dd8b7c7fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.220471 10327 tablet_service.cc:1467] Processing CreateTablet for tablet a8fba2019b474ce4b80d6b6d7c81a2ff (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240507 08:05:11.221705 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fba2019b474ce4b80d6b6d7c81a2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.221194 10381 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:11.226055 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19878aa675724839ac9bb59333dee2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.245234 10248 tablet_service.cc:1467] Processing CreateTablet for tablet e9adf1535c974b2abf910f3c1c537627 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240507 08:05:11.246181 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9800706282e543be9f2ee3473da2b89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.246834 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9adf1535c974b2abf910f3c1c537627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.247431 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3df41f82bc854e7c9a22f3f345b03f92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.248535 10175 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd78797b4574a049447c358499ad846 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240507 08:05:11.249943 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd78797b4574a049447c358499ad846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.230357 10383 tablet_bootstrap.cc:492] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:11.230186 10173 tablet_service.cc:1467] Processing CreateTablet for tablet e9adf1535c974b2abf910f3c1c537627 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240507 08:05:11.253050 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9adf1535c974b2abf910f3c1c537627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.231778 10250 tablet_service.cc:1467] Processing CreateTablet for tablet a8fba2019b474ce4b80d6b6d7c81a2ff (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20240507 08:05:11.254063 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8fba2019b474ce4b80d6b6d7c81a2ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.236423 10329 tablet_service.cc:1467] Processing CreateTablet for tablet e9adf1535c974b2abf910f3c1c537627 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20240507 08:05:11.222896 10382 tablet_bootstrap.cc:492] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:11.263258 10382 ts_tablet_manager.cc:1367] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.205s user 0.017s sys 0.002s
I20240507 08:05:11.270143 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9adf1535c974b2abf910f3c1c537627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.271349 10330 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd78797b4574a049447c358499ad846 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240507 08:05:11.271945 10382 raft_consensus.cc:348] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.274255 10382 raft_consensus.cc:374] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:11.274580 10382 raft_consensus.cc:729] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:11.272818 10325 tablet_service.cc:1467] Processing CreateTablet for tablet 3490f6d2f719474bb4f72a5647a9a726 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240507 08:05:11.272917 10324 tablet_service.cc:1467] Processing CreateTablet for tablet 324b71380c6c47bbb335661d299baea0 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240507 08:05:11.274438 10180 tablet_service.cc:1467] Processing CreateTablet for tablet 324b71380c6c47bbb335661d299baea0 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240507 08:05:11.276770 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324b71380c6c47bbb335661d299baea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.276731 10382 consensus_queue.cc:260] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.278925 10324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324b71380c6c47bbb335661d299baea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.281863 10325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3490f6d2f719474bb4f72a5647a9a726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.282860 10177 tablet_service.cc:1467] Processing CreateTablet for tablet c6ac20a6d50a4700a577dfc18d71ceb2 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240507 08:05:11.283414 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd78797b4574a049447c358499ad846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.284170 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac20a6d50a4700a577dfc18d71ceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.288515 10178 tablet_service.cc:1467] Processing CreateTablet for tablet 3490f6d2f719474bb4f72a5647a9a726 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240507 08:05:11.296830 10381 ts_tablet_manager.cc:1398] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.101s user 0.008s sys 0.001s
I20240507 08:05:11.297672 10366 heartbeater.cc:503] Master 127.8.213.190:37115 was elected leader, sending a full tablet report...
I20240507 08:05:11.297940 10381 tablet_bootstrap.cc:492] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:11.307870 10383 tablet_bootstrap.cc:654] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.311086 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3490f6d2f719474bb4f72a5647a9a726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.315337 10256 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd78797b4574a049447c358499ad846 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20240507 08:05:11.316628 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd78797b4574a049447c358499ad846. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.319532 10249 tablet_service.cc:1467] Processing CreateTablet for tablet 324b71380c6c47bbb335661d299baea0 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20240507 08:05:11.325613 10326 tablet_service.cc:1467] Processing CreateTablet for tablet c6ac20a6d50a4700a577dfc18d71ceb2 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240507 08:05:11.325762 10251 tablet_service.cc:1467] Processing CreateTablet for tablet 7a181f4ae4d342129ad262e5545debc9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20240507 08:05:11.326846 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac20a6d50a4700a577dfc18d71ceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.325852 10177 tablet_service.cc:1467] Processing CreateTablet for tablet d3f101fe713f47b1b84bcaeb54f7aead (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240507 08:05:11.338682 10174 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2e621edea41ae97e79e36e4dc0365 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240507 08:05:11.338639 10382 ts_tablet_manager.cc:1398] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.075s user 0.009s sys 0.006s
I20240507 08:05:11.345759 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324b71380c6c47bbb335661d299baea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.356448 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 8933e79c6c78466498a6d17d93ce7cff (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240507 08:05:11.356691 10255 tablet_service.cc:1467] Processing CreateTablet for tablet 3490f6d2f719474bb4f72a5647a9a726 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20240507 08:05:11.358397 10217 heartbeater.cc:503] Master 127.8.213.190:37115 was elected leader, sending a full tablet report...
I20240507 08:05:11.358958 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2e621edea41ae97e79e36e4dc0365. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:05:11.363961 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:11.364425 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a181f4ae4d342129ad262e5545debc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.364830 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8933e79c6c78466498a6d17d93ce7cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.365931 10382 tablet_bootstrap.cc:492] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:11.377379 10328 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2e621edea41ae97e79e36e4dc0365 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240507 08:05:11.378530 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f101fe713f47b1b84bcaeb54f7aead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.378731 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2e621edea41ae97e79e36e4dc0365. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.380919 10252 tablet_service.cc:1467] Processing CreateTablet for tablet 03d2e621edea41ae97e79e36e4dc0365 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20240507 08:05:11.382119 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2e621edea41ae97e79e36e4dc0365. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.382995 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3490f6d2f719474bb4f72a5647a9a726. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.378392 10248 tablet_service.cc:1467] Processing CreateTablet for tablet d3f101fe713f47b1b84bcaeb54f7aead (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240507 08:05:11.387084 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f101fe713f47b1b84bcaeb54f7aead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.354473 10381 tablet_bootstrap.cc:654] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.397701 10253 tablet_service.cc:1467] Processing CreateTablet for tablet 971f59883ca347648664bc7f2fcaf773 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240507 08:05:11.385180 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 8933e79c6c78466498a6d17d93ce7cff (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240507 08:05:11.399693 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971f59883ca347648664bc7f2fcaf773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.394631 10257 tablet_service.cc:1467] Processing CreateTablet for tablet c6ac20a6d50a4700a577dfc18d71ceb2 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20240507 08:05:11.419528 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8933e79c6c78466498a6d17d93ce7cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.420151 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac20a6d50a4700a577dfc18d71ceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.407519 10382 tablet_bootstrap.cc:654] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.412796 10179 tablet_service.cc:1467] Processing CreateTablet for tablet 971f59883ca347648664bc7f2fcaf773 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240507 08:05:11.427765 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971f59883ca347648664bc7f2fcaf773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.413933 10331 tablet_service.cc:1467] Processing CreateTablet for tablet d3f101fe713f47b1b84bcaeb54f7aead (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20240507 08:05:11.415104 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 8933e79c6c78466498a6d17d93ce7cff (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20240507 08:05:11.416137 10383 tablet_bootstrap.cc:492] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:11.435724 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f101fe713f47b1b84bcaeb54f7aead. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.436340 10383 ts_tablet_manager.cc:1367] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.206s user 0.000s sys 0.017s
W20240507 08:05:11.445168 10367 tablet.cc:2369] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:05:11.463611 10383 raft_consensus.cc:348] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.464545 10383 raft_consensus.cc:374] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:11.464908 10383 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:11.470077 10383 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.474408 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8933e79c6c78466498a6d17d93ce7cff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.519052 10180 tablet_service.cc:1467] Processing CreateTablet for tablet e0a774a8075d4c689b047bb943c78c50 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240507 08:05:11.522521 10292 heartbeater.cc:503] Master 127.8.213.190:37115 was elected leader, sending a full tablet report...
I20240507 08:05:11.523649 10175 tablet_service.cc:1467] Processing CreateTablet for tablet e57f967aaaf34bd5acd85d28edeb4abe (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240507 08:05:11.526592 10255 tablet_service.cc:1467] Processing CreateTablet for tablet e0a774a8075d4c689b047bb943c78c50 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240507 08:05:11.528718 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a774a8075d4c689b047bb943c78c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.519167 10330 tablet_service.cc:1467] Processing CreateTablet for tablet 971f59883ca347648664bc7f2fcaf773 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20240507 08:05:11.531920 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 971f59883ca347648664bc7f2fcaf773. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.516413 10329 tablet_service.cc:1467] Processing CreateTablet for tablet e0a774a8075d4c689b047bb943c78c50 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20240507 08:05:11.541819 10252 tablet_service.cc:1467] Processing CreateTablet for tablet e57f967aaaf34bd5acd85d28edeb4abe (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240507 08:05:11.543416 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57f967aaaf34bd5acd85d28edeb4abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.555421 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a774a8075d4c689b047bb943c78c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.573268 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57f967aaaf34bd5acd85d28edeb4abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.573122 10327 tablet_service.cc:1467] Processing CreateTablet for tablet 292ec14f14384ba1bc9ee81980e1eb2a (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240507 08:05:11.574744 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292ec14f14384ba1bc9ee81980e1eb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.574470 10324 tablet_service.cc:1467] Processing CreateTablet for tablet e0f47233c3be4fe99e097ade2aaec1e8 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240507 08:05:11.575834 10324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f47233c3be4fe99e097ade2aaec1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.576121 10325 tablet_service.cc:1467] Processing CreateTablet for tablet e57f967aaaf34bd5acd85d28edeb4abe (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20240507 08:05:11.577428 10325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e57f967aaaf34bd5acd85d28edeb4abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.582526 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0a774a8075d4c689b047bb943c78c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.588438 10249 tablet_service.cc:1467] Processing CreateTablet for tablet e0f47233c3be4fe99e097ade2aaec1e8 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240507 08:05:11.589776 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f47233c3be4fe99e097ade2aaec1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.588454 10248 tablet_service.cc:1467] Processing CreateTablet for tablet 292ec14f14384ba1bc9ee81980e1eb2a (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240507 08:05:11.601282 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292ec14f14384ba1bc9ee81980e1eb2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:05:11.602352 10076 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 85679.6us
I20240507 08:05:11.607295 10174 tablet_service.cc:1467] Processing CreateTablet for tablet 765054c26d4a4181b4fb31891d033bb1 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240507 08:05:11.608577 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765054c26d4a4181b4fb31891d033bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.616216 10256 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae706e85b0c447d98868f95b5a76a46 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240507 08:05:11.616535 10323 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae706e85b0c447d98868f95b5a76a46 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240507 08:05:11.617653 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae706e85b0c447d98868f95b5a76a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.617709 10323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae706e85b0c447d98868f95b5a76a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.626652 10383 ts_tablet_manager.cc:1398] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.190s user 0.017s sys 0.009s
I20240507 08:05:11.627060 10381 tablet_bootstrap.cc:492] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:11.627403 10382 tablet_bootstrap.cc:492] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:11.627785 10381 ts_tablet_manager.cc:1367] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.330s user 0.032s sys 0.013s
I20240507 08:05:11.628139 10383 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:11.628094 10382 ts_tablet_manager.cc:1367] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.262s user 0.012s sys 0.001s
I20240507 08:05:11.631307 10250 tablet_service.cc:1467] Processing CreateTablet for tablet ad8dd18470a542d3bec18f13b451878d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240507 08:05:11.632848 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8dd18470a542d3bec18f13b451878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.631757 10382 raft_consensus.cc:348] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.633198 10382 raft_consensus.cc:374] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:11.633689 10382 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:11.636029 10382 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.626631 10328 tablet_service.cc:1467] Processing CreateTablet for tablet 765054c26d4a4181b4fb31891d033bb1 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240507 08:05:11.639411 10181 tablet_service.cc:1467] Processing CreateTablet for tablet bcbe630c1736458898143fb230a9ffb8 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240507 08:05:11.639405 10326 tablet_service.cc:1467] Processing CreateTablet for tablet bcbe630c1736458898143fb230a9ffb8 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240507 08:05:11.641311 10391 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:11.641366 10387 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:11.640690 10176 tablet_service.cc:1467] Processing CreateTablet for tablet b909813a654d41cf88fb9fee2f8006bc (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240507 08:05:11.641957 10391 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.642074 10387 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.642956 10177 tablet_service.cc:1467] Processing CreateTablet for tablet ad8dd18470a542d3bec18f13b451878d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240507 08:05:11.643880 10179 tablet_service.cc:1467] Processing CreateTablet for tablet 2a210876eb5b4a7ab6548839daccfe84 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240507 08:05:11.640722 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe630c1736458898143fb230a9ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.645321 10173 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae706e85b0c447d98868f95b5a76a46 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20240507 08:05:11.646656 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae706e85b0c447d98868f95b5a76a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.649879 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909813a654d41cf88fb9fee2f8006bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.650502 10391 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:11.654145 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a210876eb5b4a7ab6548839daccfe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.667428 10387 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:11.682113 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8dd18470a542d3bec18f13b451878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.656677 10175 tablet_service.cc:1467] Processing CreateTablet for tablet 292ec14f14384ba1bc9ee81980e1eb2a (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20240507 08:05:11.683207 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 292ec14f14384ba1bc9ee81980e1eb2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.686061 10178 tablet_service.cc:1467] Processing CreateTablet for tablet e0f47233c3be4fe99e097ade2aaec1e8 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20240507 08:05:11.687353 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f47233c3be4fe99e097ade2aaec1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.692063 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe630c1736458898143fb230a9ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:05:11.694103 10076 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 52711.5us
I20240507 08:05:11.696311 10386 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:11.696925 10386 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:11.698004 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765054c26d4a4181b4fb31891d033bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.699599 10386 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:11.698020 10381 raft_consensus.cc:348] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.711745 10381 raft_consensus.cc:374] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:11.712071 10381 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:11.712796 10381 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.714813 10382 ts_tablet_manager.cc:1398] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.086s user 0.007s sys 0.001s
I20240507 08:05:11.715999 10382 tablet_bootstrap.cc:492] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:11.722484 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 765054c26d4a4181b4fb31891d033bb1 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20240507 08:05:11.724109 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 765054c26d4a4181b4fb31891d033bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.730074 10381 ts_tablet_manager.cc:1398] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.102s user 0.010s sys 0.003s
I20240507 08:05:11.731186 10381 tablet_bootstrap.cc:492] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:11.732031 10330 tablet_service.cc:1467] Processing CreateTablet for tablet 2a210876eb5b4a7ab6548839daccfe84 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240507 08:05:11.751199 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a210876eb5b4a7ab6548839daccfe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.763089 10255 tablet_service.cc:1467] Processing CreateTablet for tablet b909813a654d41cf88fb9fee2f8006bc (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240507 08:05:11.764392 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909813a654d41cf88fb9fee2f8006bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.764890 10382 tablet_bootstrap.cc:654] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.746523 10383 tablet_bootstrap.cc:654] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.737001 10324 tablet_service.cc:1467] Processing CreateTablet for tablet b909813a654d41cf88fb9fee2f8006bc (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20240507 08:05:11.733959 10322 tablet_service.cc:1467] Processing CreateTablet for tablet ad8dd18470a542d3bec18f13b451878d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20240507 08:05:11.771946 10324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b909813a654d41cf88fb9fee2f8006bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.778959 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad8dd18470a542d3bec18f13b451878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.782403 10256 tablet_service.cc:1467] Processing CreateTablet for tablet 2a210876eb5b4a7ab6548839daccfe84 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20240507 08:05:11.783697 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a210876eb5b4a7ab6548839daccfe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.785261 10249 tablet_service.cc:1467] Processing CreateTablet for tablet 5d0e1a7b6db34cea8e152b755b52aa96 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240507 08:05:11.785708 10253 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6f917a0bed4f6289c1abc13f6260b9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240507 08:05:11.786623 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d0e1a7b6db34cea8e152b755b52aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.787231 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6f917a0bed4f6289c1abc13f6260b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.806748 10257 tablet_service.cc:1467] Processing CreateTablet for tablet f0778bf2d68148c495412379f7818b28 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240507 08:05:11.808149 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0778bf2d68148c495412379f7818b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.809819 10381 tablet_bootstrap.cc:654] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:11.835353 10251 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4b60e5e6294639a468641f215329a5 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240507 08:05:11.837177 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b60e5e6294639a468641f215329a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.846374 10252 tablet_service.cc:1467] Processing CreateTablet for tablet bcbe630c1736458898143fb230a9ffb8 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20240507 08:05:11.847680 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe630c1736458898143fb230a9ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.854852 10250 tablet_service.cc:1467] Processing CreateTablet for tablet 534af8458dc14ec49ca7308cf02b685c (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240507 08:05:11.856269 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534af8458dc14ec49ca7308cf02b685c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.857867 10386 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:11.858368 10386 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:11.860284 10386 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:11.878942 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:11.881078 10155 leader_election.cc:341] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:11.885021 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:11.887039 10156 leader_election.cc:341] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:11.887481 10156 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:11.889436 10181 tablet_service.cc:1467] Processing CreateTablet for tablet 5d0e1a7b6db34cea8e152b755b52aa96 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240507 08:05:11.890766 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d0e1a7b6db34cea8e152b755b52aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.893795 10387 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:11.903955 10179 tablet_service.cc:1467] Processing CreateTablet for tablet 534af8458dc14ec49ca7308cf02b685c (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240507 08:05:11.904513 10180 tablet_service.cc:1467] Processing CreateTablet for tablet a34da9500ed541909dbdbcc3be70a067 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240507 08:05:11.905242 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534af8458dc14ec49ca7308cf02b685c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.904958 10177 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6f917a0bed4f6289c1abc13f6260b9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240507 08:05:11.906361 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6f917a0bed4f6289c1abc13f6260b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.907143 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 63ae3c4ecd5e46ec9b5cf741db56c101 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240507 08:05:11.908313 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ae3c4ecd5e46ec9b5cf741db56c101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.906085 10174 tablet_service.cc:1467] Processing CreateTablet for tablet f0778bf2d68148c495412379f7818b28 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240507 08:05:11.909224 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0778bf2d68148c495412379f7818b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.911108 10175 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4b60e5e6294639a468641f215329a5 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240507 08:05:11.912469 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b60e5e6294639a468641f215329a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.929023 10182 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7d05e73be847e99048092c9570faed (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240507 08:05:11.913506 10248 tablet_service.cc:1467] Processing CreateTablet for tablet 63ae3c4ecd5e46ec9b5cf741db56c101 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240507 08:05:11.930750 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7d05e73be847e99048092c9570faed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.931147 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ae3c4ecd5e46ec9b5cf741db56c101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:11.959244 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34da9500ed541909dbdbcc3be70a067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.006934 10381 tablet_bootstrap.cc:492] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.007465 10381 ts_tablet_manager.cc:1367] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.277s user 0.028s sys 0.027s
I20240507 08:05:12.007243 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.008991 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.009701 10191 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
W20240507 08:05:12.010984 10306 leader_election.cc:341] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.012138 10306 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:12.012984 10386 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.013306 10386 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.013654 10386 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.024925 10383 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.025424 10383 ts_tablet_manager.cc:1367] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.398s user 0.034s sys 0.040s
I20240507 08:05:12.028895 10257 tablet_service.cc:1467] Processing CreateTablet for tablet a34da9500ed541909dbdbcc3be70a067 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240507 08:05:12.030270 10386 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.031957 10255 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7d05e73be847e99048092c9570faed (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240507 08:05:12.035224 10326 tablet_service.cc:1467] Processing CreateTablet for tablet 5d0e1a7b6db34cea8e152b755b52aa96 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20240507 08:05:12.090510 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34da9500ed541909dbdbcc3be70a067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.039404 10328 tablet_service.cc:1467] Processing CreateTablet for tablet 1f4b60e5e6294639a468641f215329a5 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20240507 08:05:12.041496 10325 tablet_service.cc:1467] Processing CreateTablet for tablet 63ae3c4ecd5e46ec9b5cf741db56c101 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20240507 08:05:12.041260 10323 tablet_service.cc:1467] Processing CreateTablet for tablet 534af8458dc14ec49ca7308cf02b685c (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20240507 08:05:12.043257 10381 raft_consensus.cc:348] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.094305 10381 raft_consensus.cc:374] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.094642 10381 raft_consensus.cc:729] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.095386 10381 consensus_queue.cc:260] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.043522 10329 tablet_service.cc:1467] Processing CreateTablet for tablet a34da9500ed541909dbdbcc3be70a067 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20240507 08:05:12.045058 10327 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7d05e73be847e99048092c9570faed (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20240507 08:05:12.047935 10173 tablet_service.cc:1467] Processing CreateTablet for tablet aa60df83ab754c7e9229c7775883d22d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240507 08:05:12.097532 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa60df83ab754c7e9229c7775883d22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.099146 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:12.099772 10192 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.048602 10330 tablet_service.cc:1467] Processing CreateTablet for tablet d26fdad3816d470a8a7537d14838ac68 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240507 08:05:12.049204 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.050057 10383 raft_consensus.cc:348] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.103523 10383 raft_consensus.cc:374] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.103843 10383 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.104022 10255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7d05e73be847e99048092c9570faed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.104563 10383 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.106846 10383 ts_tablet_manager.cc:1398] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.081s user 0.006s sys 0.000s
I20240507 08:05:12.107394 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:12.051592 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.052203 10174 tablet_service.cc:1467] Processing CreateTablet for tablet d26fdad3816d470a8a7537d14838ac68 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
W20240507 08:05:12.108932 10304 leader_election.cc:341] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.040468 10253 tablet_service.cc:1467] Processing CreateTablet for tablet 95da5fd11446429481eb60f5a0c8158e (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240507 08:05:12.110102 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26fdad3816d470a8a7537d14838ac68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.054538 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.110536 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95da5fd11446429481eb60f5a0c8158e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.060670 10254 tablet_service.cc:1467] Processing CreateTablet for tablet aa60df83ab754c7e9229c7775883d22d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240507 08:05:12.111567 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa60df83ab754c7e9229c7775883d22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.053112 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:12.113452 10304 leader_election.cc:341] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.066470 10322 tablet_service.cc:1467] Processing CreateTablet for tablet 7e6f917a0bed4f6289c1abc13f6260b9 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20240507 08:05:12.071774 10256 tablet_service.cc:1467] Processing CreateTablet for tablet d26fdad3816d470a8a7537d14838ac68 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20240507 08:05:12.115470 10256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26fdad3816d470a8a7537d14838ac68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.062956 10382 tablet_bootstrap.cc:492] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.075093 10249 tablet_service.cc:1467] Processing CreateTablet for tablet 6e83f7f3abec42d2bc76c5648cfd0aa3 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240507 08:05:12.075651 10324 tablet_service.cc:1467] Processing CreateTablet for tablet 95da5fd11446429481eb60f5a0c8158e (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240507 08:05:12.120311 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e83f7f3abec42d2bc76c5648cfd0aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.085783 10178 tablet_service.cc:1467] Processing CreateTablet for tablet 95da5fd11446429481eb60f5a0c8158e (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20240507 08:05:12.122195 10178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95da5fd11446429481eb60f5a0c8158e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.090631 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d0e1a7b6db34cea8e152b755b52aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.125852 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b60e5e6294639a468641f215329a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.037554 10331 tablet_service.cc:1467] Processing CreateTablet for tablet f0778bf2d68148c495412379f7818b28 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20240507 08:05:12.129647 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0778bf2d68148c495412379f7818b28. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:05:12.130139 10304 leader_election.cc:341] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.130353 10324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95da5fd11446429481eb60f5a0c8158e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.130594 10304 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.133893 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34da9500ed541909dbdbcc3be70a067. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.134496 10323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534af8458dc14ec49ca7308cf02b685c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.135666 10182 tablet_service.cc:1467] Processing CreateTablet for tablet 7a123878203f45319def3c4246b9fe30 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240507 08:05:12.137327 10192 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:12.138180 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d26fdad3816d470a8a7537d14838ac68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.139017 10306 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.141995 10325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ae3c4ecd5e46ec9b5cf741db56c101. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.141665 10252 tablet_service.cc:1467] Processing CreateTablet for tablet 5a168a24022a4678bd74fa028a765700 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240507 08:05:12.142889 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a168a24022a4678bd74fa028a765700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.146067 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7d05e73be847e99048092c9570faed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.148283 10177 tablet_service.cc:1467] Processing CreateTablet for tablet cd9cd8b1faef46a9bbbee733353fd123 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240507 08:05:12.089300 10179 tablet_service.cc:1467] Processing CreateTablet for tablet e940ba6acb8d4782a4c921b50b882f91 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240507 08:05:12.092056 10250 tablet_service.cc:1467] Processing CreateTablet for tablet cd9cd8b1faef46a9bbbee733353fd123 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240507 08:05:12.155874 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9cd8b1faef46a9bbbee733353fd123. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:05:12.110004 10231 leader_election.cc:341] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:12.157824 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 47952.6us
I20240507 08:05:12.088776 10251 tablet_service.cc:1467] Processing CreateTablet for tablet 7a123878203f45319def3c4246b9fe30 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240507 08:05:12.153167 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6f917a0bed4f6289c1abc13f6260b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.091050 10386 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:12.159397 10383 tablet_bootstrap.cc:492] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.159627 10386 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.160264 10386 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:12.160674 10386 raft_consensus.cc:686] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:12.161536 10386 consensus_queue.cc:237] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.166666 10176 tablet_service.cc:1467] Processing CreateTablet for tablet 6e83f7f3abec42d2bc76c5648cfd0aa3 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240507 08:05:12.169165 10175 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdcc48b264f4a9eb3eabce20e5bbd6c (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240507 08:05:12.072440 10248 tablet_service.cc:1467] Processing CreateTablet for tablet e940ba6acb8d4782a4c921b50b882f91 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240507 08:05:12.158937 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a123878203f45319def3c4246b9fe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.170746 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940ba6acb8d4782a4c921b50b882f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.125869 10382 ts_tablet_manager.cc:1367] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.410s user 0.008s sys 0.020s
W20240507 08:05:12.174499 10232 leader_election.cc:341] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.174963 10232 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:12.176061 10391 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.153985 10180 tablet_service.cc:1467] Processing CreateTablet for tablet 5a168a24022a4678bd74fa028a765700 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240507 08:05:12.185546 10175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdcc48b264f4a9eb3eabce20e5bbd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.188849 10180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a168a24022a4678bd74fa028a765700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.189473 10257 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdcc48b264f4a9eb3eabce20e5bbd6c (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240507 08:05:12.190781 10257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdcc48b264f4a9eb3eabce20e5bbd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.196748 10383 tablet_bootstrap.cc:654] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.202201 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e83f7f3abec42d2bc76c5648cfd0aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.176915 10382 raft_consensus.cc:348] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.205765 10382 raft_consensus.cc:374] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.206104 10382 raft_consensus.cc:729] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.206845 10382 consensus_queue.cc:260] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.217092 10322 tablet_service.cc:1467] Processing CreateTablet for tablet aa60df83ab754c7e9229c7775883d22d (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20240507 08:05:12.228924 10322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa60df83ab754c7e9229c7775883d22d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:05:12.230221 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 100121us
W20240507 08:05:12.241734 10074 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 106032us
I20240507 08:05:12.247455 10181 tablet_service.cc:1467] Processing CreateTablet for tablet 84da12b367b046b7bcfb08aaa6e1c97b (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240507 08:05:12.233822 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a123878203f45319def3c4246b9fe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.250375 10331 tablet_service.cc:1467] Processing CreateTablet for tablet e940ba6acb8d4782a4c921b50b882f91 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20240507 08:05:12.253752 10324 tablet_service.cc:1467] Processing CreateTablet for tablet 7a123878203f45319def3c4246b9fe30 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20240507 08:05:12.254312 10328 tablet_service.cc:1467] Processing CreateTablet for tablet cd9cd8b1faef46a9bbbee733353fd123 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20240507 08:05:12.248987 10326 tablet_service.cc:1467] Processing CreateTablet for tablet 6e83f7f3abec42d2bc76c5648cfd0aa3 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20240507 08:05:12.263334 10254 tablet_service.cc:1467] Processing CreateTablet for tablet 84da12b367b046b7bcfb08aaa6e1c97b (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240507 08:05:12.263729 10250 tablet_service.cc:1467] Processing CreateTablet for tablet 85b6fa5360054f32af996ec68d175584 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240507 08:05:12.264674 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9cd8b1faef46a9bbbee733353fd123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.265350 10252 tablet_service.cc:1467] Processing CreateTablet for tablet bfb70b9953fd486ca5f88b5999dd26a0 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240507 08:05:12.277896 10179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940ba6acb8d4782a4c921b50b882f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.276232 10328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9cd8b1faef46a9bbbee733353fd123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.289873 10326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e83f7f3abec42d2bc76c5648cfd0aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.290086 10181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da12b367b046b7bcfb08aaa6e1c97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.290644 10173 tablet_service.cc:1467] Processing CreateTablet for tablet 85b6fa5360054f32af996ec68d175584 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20240507 08:05:12.291271 10382 ts_tablet_manager.cc:1398] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.110s user 0.006s sys 0.001s
I20240507 08:05:12.276995 10254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da12b367b046b7bcfb08aaa6e1c97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.295295 10174 tablet_service.cc:1467] Processing CreateTablet for tablet bfb70b9953fd486ca5f88b5999dd26a0 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240507 08:05:12.296532 10174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb70b9953fd486ca5f88b5999dd26a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.297875 10250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b6fa5360054f32af996ec68d175584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.301862 10252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb70b9953fd486ca5f88b5999dd26a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.302201 10331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940ba6acb8d4782a4c921b50b882f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.303903 10324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a123878203f45319def3c4246b9fe30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.320755 10383 tablet_bootstrap.cc:492] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.321345 10383 ts_tablet_manager.cc:1367] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.200s user 0.026s sys 0.009s
I20240507 08:05:12.274947 10330 tablet_service.cc:1467] Processing CreateTablet for tablet 5a168a24022a4678bd74fa028a765700 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20240507 08:05:12.306398 10382 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.322641 10325 tablet_service.cc:1467] Processing CreateTablet for tablet 84da12b367b046b7bcfb08aaa6e1c97b (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20240507 08:05:12.323122 10330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a168a24022a4678bd74fa028a765700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.323920 10325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da12b367b046b7bcfb08aaa6e1c97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.324046 10383 raft_consensus.cc:348] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.324805 10383 raft_consensus.cc:374] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.325129 10383 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.326316 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b6fa5360054f32af996ec68d175584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.330531 10323 tablet_service.cc:1467] Processing CreateTablet for tablet 4cdcc48b264f4a9eb3eabce20e5bbd6c (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20240507 08:05:12.331821 10323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cdcc48b264f4a9eb3eabce20e5bbd6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.307562 10381 ts_tablet_manager.cc:1398] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.300s user 0.012s sys 0.012s
I20240507 08:05:12.335389 10381 tablet_bootstrap.cc:492] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.341127 10381 tablet_bootstrap.cc:654] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.342908 10327 tablet_service.cc:1467] Processing CreateTablet for tablet bfb70b9953fd486ca5f88b5999dd26a0 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20240507 08:05:12.374408 10327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfb70b9953fd486ca5f88b5999dd26a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.347229 10329 tablet_service.cc:1467] Processing CreateTablet for tablet 85b6fa5360054f32af996ec68d175584 (DEFAULT_TABLE table=test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20240507 08:05:12.282963 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 144029us
I20240507 08:05:12.344833 10407 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:12.366562 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:12.373890 10383 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.353106 10382 tablet_bootstrap.cc:654] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.375986 10329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b6fa5360054f32af996ec68d175584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:12.375757 10407 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.381523 10407 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:12.388573 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.389278 10192 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:12.389775 10383 ts_tablet_manager.cc:1398] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.068s user 0.010s sys 0.000s
I20240507 08:05:12.392035 10306 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:12.392868 10407 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.393203 10407 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.398432 10407 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.401650 10090 catalog_manager.cc:5484] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:12.405097 10382 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.406687 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.407593 10383 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
W20240507 08:05:12.415478 10304 leader_election.cc:341] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.423417 10383 tablet_bootstrap.cc:654] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.424072 10382 ts_tablet_manager.cc:1367] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.118s user 0.011s sys 0.013s
I20240507 08:05:12.428491 10382 raft_consensus.cc:348] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.431679 10382 raft_consensus.cc:374] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.432787 10382 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.433888 10382 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.436875 10407 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.441488 10382 ts_tablet_manager.cc:1398] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20240507 08:05:12.439898 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:12.442338 10381 tablet_bootstrap.cc:492] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.442499 10382 tablet_bootstrap.cc:492] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.442668 10192 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.442795 10381 ts_tablet_manager.cc:1367] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.108s user 0.023s sys 0.026s
I20240507 08:05:12.445519 10381 raft_consensus.cc:348] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.446219 10381 raft_consensus.cc:374] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.446522 10381 raft_consensus.cc:729] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.447222 10381 consensus_queue.cc:260] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.448213 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:12.448803 10382 tablet_bootstrap.cc:654] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.449440 10381 ts_tablet_manager.cc:1398] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:12.448702 10407 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:12.449860 10304 leader_election.cc:341] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.450560 10381 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.449469 10383 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.451090 10383 ts_tablet_manager.cc:1367] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.003s
I20240507 08:05:12.454473 10391 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.454921 10391 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.455498 10192 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:12.456919 10391 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.456858 10306 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.458748 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:12.460314 10232 leader_election.cc:341] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.460574 10381 tablet_bootstrap.cc:654] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.458667 10383 raft_consensus.cc:348] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.461369 10383 raft_consensus.cc:374] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.462307 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.463483 10391 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.464004 10407 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:12.463932 10391 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.464692 10407 raft_consensus.cc:686] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:12.465950 10391 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.465534 10407 consensus_queue.cc:237] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:12.466899 10231 leader_election.cc:341] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.467335 10231 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:12.462328 10383 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.468566 10383 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.470045 10391 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.470213 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.471067 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.471836 10192 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:12.472822 10232 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:12.478482 10383 ts_tablet_manager.cc:1398] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.025s user 0.006s sys 0.002s
I20240507 08:05:12.479779 10383 tablet_bootstrap.cc:492] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.491430 10418 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.493722 10383 tablet_bootstrap.cc:654] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.502311 10418 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.512506 10382 tablet_bootstrap.cc:492] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.513109 10382 ts_tablet_manager.cc:1367] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.071s user 0.049s sys 0.015s
I20240507 08:05:12.512534 10381 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.515151 10387 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.515341 10381 ts_tablet_manager.cc:1367] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.064s user 0.021s sys 0.025s
I20240507 08:05:12.515664 10387 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.516700 10382 raft_consensus.cc:348] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.517568 10382 raft_consensus.cc:374] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.517813 10387 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.517944 10382 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.516840 10088 catalog_manager.cc:5484] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:12.519460 10382 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.520346 10381 raft_consensus.cc:348] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.520784 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.521075 10381 raft_consensus.cc:374] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.519927 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.521445 10341 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:12.523707 10156 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:12.521399 10381 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.525259 10382 ts_tablet_manager.cc:1398] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240507 08:05:12.525077 10381 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.526235 10387 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240507 08:05:12.525978 10155 leader_election.cc:341] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.526613 10387 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.526942 10387 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.527997 10381 ts_tablet_manager.cc:1398] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.012s user 0.005s sys 0.001s
I20240507 08:05:12.528978 10381 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.529737 10382 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.534423 10381 tablet_bootstrap.cc:654] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.534093 10387 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.537509 10382 tablet_bootstrap.cc:654] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.537292 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:12.537940 10341 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.540683 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:12.542563 10155 leader_election.cc:341] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.547040 10387 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.548331 10341 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:12.549810 10156 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.551265 10387 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:12.554657 10387 raft_consensus.cc:686] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:12.555778 10387 consensus_queue.cc:237] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.564252 10383 tablet_bootstrap.cc:492] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.564201 10382 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.565030 10382 ts_tablet_manager.cc:1367] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20240507 08:05:12.565111 10383 ts_tablet_manager.cc:1367] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.086s user 0.016s sys 0.008s
I20240507 08:05:12.566599 10381 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.567061 10381 ts_tablet_manager.cc:1367] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.038s user 0.017s sys 0.016s
I20240507 08:05:12.567840 10382 raft_consensus.cc:348] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.568593 10382 raft_consensus.cc:374] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.568907 10382 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.568665 10383 raft_consensus.cc:348] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.569520 10381 raft_consensus.cc:348] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.569738 10383 raft_consensus.cc:374] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.570253 10381 raft_consensus.cc:374] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.569742 10382 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.570660 10381 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.570340 10383 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.571404 10381 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.572736 10382 ts_tablet_manager.cc:1398] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240507 08:05:12.573760 10382 tablet_bootstrap.cc:492] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.574653 10381 ts_tablet_manager.cc:1398] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240507 08:05:12.571856 10383 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.576825 10383 ts_tablet_manager.cc:1398] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240507 08:05:12.577888 10383 tablet_bootstrap.cc:492] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.578011 10381 tablet_bootstrap.cc:492] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.579887 10088 catalog_manager.cc:5484] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:12.583007 10382 tablet_bootstrap.cc:654] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.584129 10383 tablet_bootstrap.cc:654] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.584196 10381 tablet_bootstrap.cc:654] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.633467 10383 tablet_bootstrap.cc:492] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.633961 10383 ts_tablet_manager.cc:1367] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.056s user 0.027s sys 0.027s
I20240507 08:05:12.634428 10381 tablet_bootstrap.cc:492] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.634418 10382 tablet_bootstrap.cc:492] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.635006 10381 ts_tablet_manager.cc:1367] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.057s user 0.035s sys 0.021s
I20240507 08:05:12.635035 10382 ts_tablet_manager.cc:1367] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.061s user 0.021s sys 0.023s
I20240507 08:05:12.636343 10383 raft_consensus.cc:348] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.636984 10383 raft_consensus.cc:374] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.637320 10383 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.637822 10381 raft_consensus.cc:348] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.638167 10383 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.638063 10382 raft_consensus.cc:348] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.638587 10381 raft_consensus.cc:374] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.638911 10418 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.638952 10382 raft_consensus.cc:374] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.638967 10381 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.639547 10382 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.639487 10418 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.639963 10381 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.640440 10382 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.642318 10381 ts_tablet_manager.cc:1398] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240507 08:05:12.643090 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.643894 10381 tablet_bootstrap.cc:492] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.644037 10383 ts_tablet_manager.cc:1398] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240507 08:05:12.644958 10383 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.646008 10382 ts_tablet_manager.cc:1398] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240507 08:05:12.646965 10382 tablet_bootstrap.cc:492] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.650375 10381 tablet_bootstrap.cc:654] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.652782 10382 tablet_bootstrap.cc:654] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.643808 10192 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:12.650311 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.654269 10341 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:12.654419 10232 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.655211 10391 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.655560 10391 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.655926 10383 tablet_bootstrap.cc:654] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.655936 10391 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.657895 10387 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.658355 10387 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.660917 10387 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.662994 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.663877 10267 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:12.666090 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.666750 10341 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:12.667958 10383 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.668172 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.668428 10383 ts_tablet_manager.cc:1367] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20240507 08:05:12.671103 10383 raft_consensus.cc:348] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.671829 10383 raft_consensus.cc:374] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.672119 10383 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.677515 10407 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:12.681059 10426 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.681561 10426 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.681974 10426 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.682579 10418 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.688109 10387 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.688593 10387 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.689064 10407 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:12.691344 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:12.692816 10155 leader_election.cc:341] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.695137 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:12.696391 10156 leader_election.cc:341] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.696838 10156 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.696869 10426 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.708418 10426 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.709143 10426 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.698254 10387 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.709195 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:12.709865 10266 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.705287 10418 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.714330 10418 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.715802 10434 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.716246 10434 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.717367 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.718062 10191 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:12.719007 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.719625 10265 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:12.719872 10232 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.720587 10418 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.721071 10418 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.721194 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.721899 10341 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:12.723557 10155 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.724362 10387 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.724714 10387 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.725037 10387 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.672973 10383 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.727300 10383 ts_tablet_manager.cc:1398] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.059s user 0.006s sys 0.001s
I20240507 08:05:12.728250 10383 tablet_bootstrap.cc:492] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.721419 10418 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.700939 10391 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.729794 10418 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.731905 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:12.732491 10191 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.739933 10191 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:12.739792 10387 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.741060 10232 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.741894 10435 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:12.742561 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:12.746037 10266 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:12.747123 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.748020 10426 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:12.748543 10426 raft_consensus.cc:686] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:12.749347 10391 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.750511 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:12.751119 10341 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:12.753829 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 57469us
I20240507 08:05:12.755587 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:12.756201 10340 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.757650 10426 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.763496 10340 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:12.765738 10341 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:12.770051 10434 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.770882 10383 tablet_bootstrap.cc:654] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.773662 10435 raft_consensus.cc:686] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:12.774547 10435 consensus_queue.cc:237] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:12.783092 10387 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.801313 10382 tablet_bootstrap.cc:492] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.801864 10382 ts_tablet_manager.cc:1367] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.155s user 0.054s sys 0.077s
I20240507 08:05:12.804471 10382 raft_consensus.cc:348] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.804306 10090 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:12.805120 10382 raft_consensus.cc:374] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.806061 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:12.807123 10382 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.808424 10382 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.809872 10340 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.815609 10418 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.817153 10265 raft_consensus.cc:2379] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:12.819152 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:12.819947 10191 raft_consensus.cc:2379] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:12.820935 10382 ts_tablet_manager.cc:1398] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.019s user 0.004s sys 0.002s
I20240507 08:05:12.821847 10382 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.821772 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:12.823979 10232 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:12.794327 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.826925 10435 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.819344 10418 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.829406 10090 catalog_manager.cc:5484] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:12.830171 10435 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.834607 10435 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.840004 10435 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.839931 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.841236 10381 tablet_bootstrap.cc:492] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.840637 10435 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.841878 10381 ts_tablet_manager.cc:1367] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.198s user 0.061s sys 0.114s
I20240507 08:05:12.842042 10435 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:12.842577 10232 leader_election.cc:341] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.839931 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.845216 10381 raft_consensus.cc:348] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.854101 10381 raft_consensus.cc:374] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.854400 10381 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.855001 10381 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.857707 10381 ts_tablet_manager.cc:1398] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
W20240507 08:05:12.857833 10231 leader_election.cc:341] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.858250 10231 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:12.858642 10381 tablet_bootstrap.cc:492] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.864145 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:12.865648 10231 leader_election.cc:341] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.866473 10435 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.867179 10435 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.867842 10340 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:12.868772 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.869017 10156 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:12.871554 10232 leader_election.cc:341] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.871961 10232 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:12.872606 10418 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.871121 10426 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:12.873415 10426 raft_consensus.cc:686] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:12.874182 10426 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.876816 10381 tablet_bootstrap.cc:654] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.879245 10382 tablet_bootstrap.cc:654] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.886664 10426 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.887315 10383 tablet_bootstrap.cc:492] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.887158 10426 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.887930 10383 ts_tablet_manager.cc:1367] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.160s user 0.010s sys 0.033s
I20240507 08:05:12.889830 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.890486 10267 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:12.890712 10090 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:12.891592 10426 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.891638 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:12.892511 10426 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:12.892575 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:12.892937 10426 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:12.893286 10426 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.893235 10340 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:12.891068 10383 raft_consensus.cc:348] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.895094 10383 raft_consensus.cc:374] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.895362 10383 raft_consensus.cc:729] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.896046 10383 consensus_queue.cc:260] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.898336 10383 ts_tablet_manager.cc:1398] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240507 08:05:12.899214 10383 tablet_bootstrap.cc:492] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.900173 10426 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.902405 10426 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:12.903482 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:12.904094 10340 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.905067 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:12.905654 10267 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:12.907002 10382 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.908514 10382 ts_tablet_manager.cc:1367] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.087s user 0.009s sys 0.005s
I20240507 08:05:12.910996 10382 raft_consensus.cc:348] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.911768 10382 raft_consensus.cc:374] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.911291 10340 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:12.912118 10382 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.913053 10156 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:12.912829 10382 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.914222 10446 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.914696 10446 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.915369 10267 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:12.915805 10383 tablet_bootstrap.cc:654] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.916807 10446 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:12.917006 10381 tablet_bootstrap.cc:492] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.917457 10381 ts_tablet_manager.cc:1367] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.009s
I20240507 08:05:12.918889 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.919526 10381 raft_consensus.cc:348] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.920007 10381 raft_consensus.cc:374] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.920202 10381 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.920807 10383 tablet_bootstrap.cc:492] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
W20240507 08:05:12.920723 10306 leader_election.cc:341] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.920670 10381 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.921329 10383 ts_tablet_manager.cc:1367] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.003s
I20240507 08:05:12.922555 10381 ts_tablet_manager.cc:1398] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240507 08:05:12.923710 10381 tablet_bootstrap.cc:492] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
W20240507 08:05:12.924019 10304 consensus_peers.cc:485] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:12.923720 10383 raft_consensus.cc:348] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.924548 10383 raft_consensus.cc:374] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.924813 10383 raft_consensus.cc:729] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.925959 10382 ts_tablet_manager.cc:1398] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20240507 08:05:12.925478 10383 consensus_queue.cc:260] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.926802 10382 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.928787 10381 tablet_bootstrap.cc:654] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.930922 10383 ts_tablet_manager.cc:1398] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240507 08:05:12.931758 10383 tablet_bootstrap.cc:492] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.935714 10382 tablet_bootstrap.cc:654] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.938217 10446 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:12.938699 10446 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.941412 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:12.941848 10383 tablet_bootstrap.cc:654] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:12.942797 10306 leader_election.cc:341] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.943790 10446 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:12.944520 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.945343 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:12.945741 10426 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:05:12.947279 10304 leader_election.cc:341] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.948446 10304 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.949632 10446 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:05:12.950318 10304 leader_election.cc:341] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:12.950994 10304 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:12.952982 10446 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:12.947245 10426 raft_consensus.cc:686] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:12.954232 10426 consensus_queue.cc:237] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.962797 10382 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.963447 10382 ts_tablet_manager.cc:1367] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.010s
I20240507 08:05:12.966380 10382 raft_consensus.cc:348] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.967221 10382 raft_consensus.cc:374] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.967469 10382 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.967773 10383 tablet_bootstrap.cc:492] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.968194 10383 ts_tablet_manager.cc:1367] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.037s user 0.014s sys 0.006s
I20240507 08:05:12.968137 10382 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.970026 10381 tablet_bootstrap.cc:492] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.970399 10382 ts_tablet_manager.cc:1398] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240507 08:05:12.970444 10381 ts_tablet_manager.cc:1367] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.047s user 0.026s sys 0.016s
I20240507 08:05:12.971364 10382 tablet_bootstrap.cc:492] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:12.970680 10090 catalog_manager.cc:5484] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:12.971823 10383 raft_consensus.cc:348] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.972587 10383 raft_consensus.cc:374] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.972890 10383 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.973300 10381 raft_consensus.cc:348] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.973965 10381 raft_consensus.cc:374] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.973520 10383 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.974241 10381 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.974938 10381 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.975786 10383 ts_tablet_manager.cc:1398] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20240507 08:05:12.976668 10383 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:12.976895 10381 ts_tablet_manager.cc:1398] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:05:12.978078 10381 tablet_bootstrap.cc:492] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:12.981962 10383 tablet_bootstrap.cc:654] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.983208 10381 tablet_bootstrap.cc:654] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.988194 10382 tablet_bootstrap.cc:654] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:12.994812 10383 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:12.996434 10383 ts_tablet_manager.cc:1367] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.001s
I20240507 08:05:12.997444 10381 tablet_bootstrap.cc:492] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:12.997886 10381 ts_tablet_manager.cc:1367] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.006s
I20240507 08:05:12.998327 10382 tablet_bootstrap.cc:492] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:12.998687 10382 ts_tablet_manager.cc:1367] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.003s
I20240507 08:05:12.998868 10383 raft_consensus.cc:348] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:12.999547 10383 raft_consensus.cc:374] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:12.999812 10383 raft_consensus.cc:729] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:12.999969 10418 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.000149 10381 raft_consensus.cc:348] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.000392 10418 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.000814 10381 raft_consensus.cc:374] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.000440 10383 consensus_queue.cc:260] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.001097 10381 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.001799 10381 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.002257 10418 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.002478 10383 ts_tablet_manager.cc:1398] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:13.003687 10381 ts_tablet_manager.cc:1398] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20240507 08:05:13.004168 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.004572 10381 tablet_bootstrap.cc:492] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.005342 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:13.006644 10232 leader_election.cc:341] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:13.007899 10231 leader_election.cc:341] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.008343 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:13.009124 10381 tablet_bootstrap.cc:654] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.009356 10418 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:13.010880 10382 raft_consensus.cc:348] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.011583 10382 raft_consensus.cc:374] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.011850 10382 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.012063 10383 tablet_bootstrap.cc:492] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:13.012529 10382 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.017342 10383 tablet_bootstrap.cc:654] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.019464 10382 ts_tablet_manager.cc:1398] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20240507 08:05:13.020646 10382 tablet_bootstrap.cc:492] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.030582 10381 tablet_bootstrap.cc:492] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:13.031046 10381 ts_tablet_manager.cc:1367] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.004s
I20240507 08:05:13.033468 10382 tablet_bootstrap.cc:654] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.033867 10381 raft_consensus.cc:348] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.034672 10381 raft_consensus.cc:374] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.034971 10381 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.035509 10381 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.036695 10383 tablet_bootstrap.cc:492] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:13.037184 10381 ts_tablet_manager.cc:1398] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:05:13.037276 10383 ts_tablet_manager.cc:1367] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20240507 08:05:13.038343 10381 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.047027 10383 raft_consensus.cc:348] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.048133 10383 raft_consensus.cc:374] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.048558 10381 tablet_bootstrap.cc:654] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.048723 10383 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.049957 10383 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.055877 10381 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:13.056380 10381 ts_tablet_manager.cc:1367] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.008s
I20240507 08:05:13.056583 10383 ts_tablet_manager.cc:1398] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20240507 08:05:13.057842 10383 tablet_bootstrap.cc:492] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:13.058991 10381 raft_consensus.cc:348] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.059767 10381 raft_consensus.cc:374] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.059988 10381 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.060508 10381 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.062918 10381 ts_tablet_manager.cc:1398] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240507 08:05:13.063838 10381 tablet_bootstrap.cc:492] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.068981 10381 tablet_bootstrap.cc:654] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.071993 10382 tablet_bootstrap.cc:492] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:13.072413 10382 ts_tablet_manager.cc:1367] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.052s user 0.033s sys 0.010s
I20240507 08:05:13.074847 10382 raft_consensus.cc:348] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.075520 10382 raft_consensus.cc:374] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.075781 10382 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.076483 10383 tablet_bootstrap.cc:654] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.076445 10382 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.086249 10383 tablet_bootstrap.cc:492] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:13.086580 10446 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.086961 10383 ts_tablet_manager.cc:1367] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.003s
I20240507 08:05:13.087496 10382 ts_tablet_manager.cc:1398] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20240507 08:05:13.090723 10383 raft_consensus.cc:348] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.093739 10383 raft_consensus.cc:374] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.094257 10383 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.095367 10383 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.097945 10382 tablet_bootstrap.cc:492] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.098696 10383 ts_tablet_manager.cc:1398] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20240507 08:05:13.099917 10383 tablet_bootstrap.cc:492] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:13.104324 10267 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.104494 10381 tablet_bootstrap.cc:492] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:13.104895 10381 ts_tablet_manager.cc:1367] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.041s user 0.022s sys 0.016s
I20240507 08:05:13.106037 10382 tablet_bootstrap.cc:654] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.107367 10381 raft_consensus.cc:348] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.108045 10381 raft_consensus.cc:374] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.108292 10381 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.109027 10381 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.113003 10426 consensus_queue.cc:1035] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.114606 10381 ts_tablet_manager.cc:1398] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20240507 08:05:13.115521 10381 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.121405 10381 tablet_bootstrap.cc:654] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.122462 10446 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.122956 10446 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.125051 10446 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.127293 10382 tablet_bootstrap.cc:492] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:13.127727 10382 ts_tablet_manager.cc:1367] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.010s
I20240507 08:05:13.128181 10383 tablet_bootstrap.cc:654] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.129426 10446 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.131289 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:13.133387 10306 leader_election.cc:341] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.130537 10382 raft_consensus.cc:348] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.134649 10382 raft_consensus.cc:374] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.135022 10382 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.136544 10382 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.129935 10446 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.138887 10382 ts_tablet_manager.cc:1398] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20240507 08:05:13.140661 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.141287 10192 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.143920 10382 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.144781 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:13.145231 10381 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:13.145666 10381 ts_tablet_manager.cc:1367] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.004s
I20240507 08:05:13.145779 10409 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.146137 10409 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.146441 10409 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.148020 10381 raft_consensus.cc:348] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.148718 10381 raft_consensus.cc:374] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.148993 10381 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.149888 10382 tablet_bootstrap.cc:654] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.149976 10381 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.151190 10418 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.151603 10418 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.152312 10381 ts_tablet_manager.cc:1398] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240507 08:05:13.153189 10381 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.154577 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.158514 10381 tablet_bootstrap.cc:654] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.153509 10418 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.162566 10383 tablet_bootstrap.cc:492] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:13.162968 10383 ts_tablet_manager.cc:1367] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.063s user 0.014s sys 0.006s
I20240507 08:05:13.164578 10409 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.165238 10383 raft_consensus.cc:348] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.166035 10383 raft_consensus.cc:374] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.166333 10383 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.167021 10383 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.168969 10383 ts_tablet_manager.cc:1398] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:05:13.169904 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.171360 10232 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:13.172215 10387 consensus_queue.cc:1035] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.173250 10446 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.173987 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.172463 10418 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:13.175251 10426 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.175251 10383 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:13.175741 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.175786 10426 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.176512 10266 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.176784 10381 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:13.177178 10381 ts_tablet_manager.cc:1367] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.000s
W20240507 08:05:13.177984 10304 leader_election.cc:341] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.178418 10304 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:13.180594 10387 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:13.181166 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.182165 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:13.182754 10156 leader_election.cc:341] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.182976 10266 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:13.183419 10426 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.184638 10155 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:13.181490 10383 tablet_bootstrap.cc:654] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.187170 10407 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:13.188659 10382 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:13.189097 10382 ts_tablet_manager.cc:1367] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.049s user 0.021s sys 0.010s
I20240507 08:05:13.189306 10381 raft_consensus.cc:348] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.190085 10381 raft_consensus.cc:374] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.190382 10381 raft_consensus.cc:729] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.191457 10382 raft_consensus.cc:348] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.192087 10382 raft_consensus.cc:374] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.192312 10382 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.192979 10382 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.193924 10434 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.194478 10434 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.193892 10381 consensus_queue.cc:260] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.194824 10434 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.198107 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:13.198855 10192 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.199838 10407 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.200270 10407 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.201504 10434 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.202222 10407 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.202723 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:13.203320 10265 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.203946 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.204339 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:13.205153 10192 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
W20240507 08:05:13.205296 10306 leader_election.cc:341] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:13.205662 10156 leader_election.cc:341] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.206704 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:13.207322 10382 ts_tablet_manager.cc:1398] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.018s user 0.003s sys 0.003s
I20240507 08:05:13.207825 10446 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:13.208178 10382 tablet_bootstrap.cc:492] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.208474 10446 raft_consensus.cc:686] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:13.209970 10446 consensus_queue.cc:237] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.212990 10434 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.215003 10382 tablet_bootstrap.cc:654] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.216048 10381 ts_tablet_manager.cc:1398] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20240507 08:05:13.217254 10381 tablet_bootstrap.cc:492] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.217953 10409 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.218343 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:13.218940 10267 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.228665 10382 tablet_bootstrap.cc:492] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:13.229189 10382 ts_tablet_manager.cc:1367] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.006s
I20240507 08:05:13.230829 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.232432 10266 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.234597 10382 raft_consensus.cc:348] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.235069 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:13.235389 10382 raft_consensus.cc:374] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.235644 10382 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.237185 10382 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.243590 10267 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:13.245167 10382 ts_tablet_manager.cc:1398] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20240507 08:05:13.246032 10382 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.246660 10265 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:13.246807 10436 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:13.251039 10446 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.257637 10446 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.258296 10446 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.252336 10382 tablet_bootstrap.cc:654] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.259708 10155 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:13.259907 10436 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.260530 10436 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.261142 10387 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:13.261683 10387 raft_consensus.cc:686] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:13.262451 10387 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.264393 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.265214 10341 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:13.265926 10436 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.266988 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.267289 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:13.276834 10383 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:13.280462 10383 ts_tablet_manager.cc:1367] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.105s user 0.056s sys 0.030s
I20240507 08:05:13.278882 10381 tablet_bootstrap.cc:654] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.281924 10446 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.284364 10387 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.284775 10387 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.285156 10387 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.285918 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:13.286505 10192 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:13.294090 10155 leader_election.cc:341] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.294171 10446 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.294417 10426 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.294904 10426 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.295573 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:13.295964 10267 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.296311 10418 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.296833 10426 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.297299 10426 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.297683 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.301689 10387 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.302597 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:13.303817 10156 leader_election.cc:341] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.306967 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:13.307412 10387 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.308913 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:13.313362 10418 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.315428 10339 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.292616 10192 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:13.316514 10383 raft_consensus.cc:348] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.317082 10306 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:13.308620 10155 leader_election.cc:341] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.315577 10418 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.318720 10383 raft_consensus.cc:374] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.320717 10383 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.321866 10383 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.330262 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.330634 10435 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.332429 10381 tablet_bootstrap.cc:492] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
W20240507 08:05:13.331526 10232 leader_election.cc:341] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.334558 10381 ts_tablet_manager.cc:1367] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.118s user 0.010s sys 0.004s
I20240507 08:05:13.333365 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.336277 10337 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:13.336900 10383 ts_tablet_manager.cc:1398] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.056s user 0.003s sys 0.004s
I20240507 08:05:13.324306 10409 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:13.339403 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:13.340696 10409 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:13.331270 10435 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.342358 10382 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:13.343024 10382 ts_tablet_manager.cc:1367] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.097s user 0.008s sys 0.017s
I20240507 08:05:13.348461 10383 tablet_bootstrap.cc:492] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:13.332187 10418 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.360881 10265 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:13.362048 10155 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:13.358400 10409 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.354084 10418 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.367323 10387 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.368927 10387 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.369370 10387 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.366887 10418 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.374783 10418 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.389791 10446 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.390592 10446 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.391861 10418 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.415347 10418 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.416487 10434 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.446223 10469 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.446815 10469 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.450346 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.417287 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:13.451798 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.450541 10469 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:13.452957 10232 leader_election.cc:341] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:13.452966 10304 leader_election.cc:341] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:13.455540 10306 leader_election.cc:341] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.458274 10426 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:13.417105 10434 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.449436 10387 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.415576 10267 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:13.562399 10469 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.562951 10469 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.660864 10381 raft_consensus.cc:348] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.661478 10381 raft_consensus.cc:374] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.661796 10381 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.519140 10089 catalog_manager.cc:5484] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:13.666370 10381 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:13.660913 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 207982us
I20240507 08:05:13.668324 10436 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240507 08:05:13.672871 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.673029 10435 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.562691 10446 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.674047 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.659092 10339 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:13.677541 10381 ts_tablet_manager.cc:1398] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.343s user 0.010s sys 0.002s
I20240507 08:05:13.677981 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.671285 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:13.620786 10306 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:13.682121 10381 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:13.683830 10436 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.684298 10436 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.685124 10387 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.685148 10382 raft_consensus.cc:348] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.685967 10382 raft_consensus.cc:374] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.686219 10436 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.686242 10382 raft_consensus.cc:729] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.686095 10435 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.687175 10382 consensus_queue.cc:260] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.681597 10446 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.690752 10382 ts_tablet_manager.cc:1398] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.347s user 0.071s sys 0.106s
I20240507 08:05:13.691622 10382 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.681413 10267 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.692119 10409 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:13.689653 10191 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:13.688247 10337 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.695950 10484 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.699049 10304 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:13.691442 10446 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.701216 10338 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
W20240507 08:05:13.701606 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:13.702324 10409 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.702551 10446 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.703475 10231 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:13.707011 10383 tablet_bootstrap.cc:654] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.707897 10469 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.712164 10382 tablet_bootstrap.cc:654] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.703683 10434 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.712599 10381 tablet_bootstrap.cc:654] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:13.695137 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 239877us
W20240507 08:05:13.703015 10370 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.IsCreateTableDone -> {remote=127.8.213.190:37115, user_credentials={real_user=slave}} blocked reactor thread for 227993us
I20240507 08:05:13.731971 10435 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.702670 10409 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.734289 10435 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.704322 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.745226 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:13.738014 10435 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.750289 10488 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:13.750767 10488 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.754048 10488 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.735783 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.738996 10409 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:13.738056 10477 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.104s user 0.001s sys 0.002s
I20240507 08:05:13.706959 10484 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.764715 10484 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.757135 10435 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.768476 10267 raft_consensus.cc:2379] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:13.769241 10435 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:13.766443 10263 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.779194 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.786466 10190 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:13.788239 10383 tablet_bootstrap.cc:492] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:13.788218 10232 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:13.788808 10383 ts_tablet_manager.cc:1367] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.443s user 0.013s sys 0.006s
W20240507 08:05:13.791128 10155 leader_election.cc:341] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.790987 10434 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.795239 10381 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:13.783948 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.796916 10483 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.793357 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.791836 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:13.780426 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:13.814750 10382 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:13.843072 10382 ts_tablet_manager.cc:1367] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.152s user 0.028s sys 0.044s
I20240507 08:05:13.841948 10381 ts_tablet_manager.cc:1367] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.160s user 0.023s sys 0.036s
I20240507 08:05:13.814163 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.844254 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.844400 10383 raft_consensus.cc:348] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.845724 10186 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.845898 10188 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.846040 10383 raft_consensus.cc:374] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.846685 10383 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.847848 10383 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.853719 10306 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:13.838939 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.854725 10306 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:13.856298 10383 ts_tablet_manager.cc:1398] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.067s user 0.025s sys 0.004s
I20240507 08:05:13.794353 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:13.842913 10483 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.857698 10446 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.863943 10192 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:13.864210 10383 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
W20240507 08:05:13.864507 10306 leader_election.cc:341] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.864835 10409 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.857841 10387 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:05:13.844758 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 65500.3us
I20240507 08:05:13.797092 10266 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:13.796461 10337 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:13.871301 10488 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:13.871327 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:13.883630 10304 leader_election.cc:341] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:13.884091 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:13.884824 10469 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:13.885391 10469 consensus_queue.cc:1035] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:13.888664 10263 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:13.882488 10488 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.890367 10488 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.890527 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:13.872562 10446 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:13.857472 10262 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.865865 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.797621 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:13.877018 10382 raft_consensus.cc:348] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.892998 10382 raft_consensus.cc:374] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:13.893296 10382 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.876029 10483 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.896256 10382 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.892823 10446 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.911082 10382 ts_tablet_manager.cc:1398] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.068s user 0.007s sys 0.000s
I20240507 08:05:13.908493 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:13.914773 10418 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:13.915261 10418 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:13.916893 10382 tablet_bootstrap.cc:492] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:13.907676 10187 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.918172 10409 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:13.907486 10387 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:13.909941 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.950212 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:13.926066 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:13.977125 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:13.916246 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:13.971975 10381 raft_consensus.cc:348] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.992542 10382 tablet_bootstrap.cc:654] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:13.919871 10469 consensus_queue.cc:1035] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.100s
I20240507 08:05:14.018352 10263 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.018045 10387 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.988078 10409 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20240507 08:05:14.018635 10335 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:14.021127 10407 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.018698 10192 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:13.949465 10446 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.998327 10483 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.952160 10488 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:13.989249 10089 catalog_manager.cc:5484] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
W20240507 08:05:13.920455 10156 leader_election.cc:341] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.006917 10187 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:14.019078 10381 raft_consensus.cc:374] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.046201 10381 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:13.969909 10418 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:14.019116 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 108213us
I20240507 08:05:14.019095 10336 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:14.028080 10382 tablet_bootstrap.cc:492] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.047389 10382 ts_tablet_manager.cc:1367] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.004s
I20240507 08:05:14.028791 10263 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:13.951431 10383 tablet_bootstrap.cc:654] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.031724 10341 raft_consensus.cc:2379] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:14.039685 10338 raft_consensus.cc:2379] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:14.040445 10488 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.042457 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:14.043747 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:14.036367 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:14.034173 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:14.059178 10155 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:14.051021 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:14.049840 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 133076us
I20240507 08:05:14.061283 10304 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:14.054611 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:14.062369 10156 consensus_peers.cc:485] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e -> Peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Couldn't send request to peer 88a26dac37fa4243b8c2b3d47f403488. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:14.050282 10483 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.035358 10446 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.059368 10188 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.051540 10381 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.059821 10484 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.068472 10469 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.056226 10192 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:14.060484 10265 raft_consensus.cc:2379] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:14.064400 10264 raft_consensus.cc:2379] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:14.063419 10232 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:14.065078 10156 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.064081 10387 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.067791 10382 raft_consensus.cc:348] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.063431 10186 raft_consensus.cc:2379] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:14.070729 10381 ts_tablet_manager.cc:1398] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.227s user 0.010s sys 0.004s
I20240507 08:05:14.071738 10499 raft_consensus.cc:2741] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240507 08:05:14.069743 10484 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.072698 10382 raft_consensus.cc:374] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.073010 10382 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.074100 10381 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.075220 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:14.077562 10483 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.077067 10382 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.078131 10435 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.075228 10503 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.078682 10483 raft_consensus.cc:686] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:14.079712 10503 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.078260 10469 raft_consensus.cc:686] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:14.080053 10503 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.077023 10387 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:14.087895 10387 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.086082 10484 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.089201 10382 ts_tablet_manager.cc:1398] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.041s user 0.001s sys 0.005s
I20240507 08:05:14.090229 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.093715 10436 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.091856 10469 consensus_queue.cc:237] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.098968 10382 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.100354 10306 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.101173 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.105129 10446 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.105324 10338 raft_consensus.cc:2379] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:14.111212 10231 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:14.114010 10499 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.114725 10484 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.114501 10499 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.116590 10499 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.080238 10483 consensus_queue.cc:237] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.119002 10090 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.122575 10090 catalog_manager.cc:5484] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.124209 10188 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:14.124745 10446 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.125249 10446 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.126184 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.126968 10338 raft_consensus.cc:2379] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:14.127418 10446 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.128893 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.138825 10484 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.142305 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.144760 10381 tablet_bootstrap.cc:654] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.143528 10156 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.154353 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.161224 10499 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.170360 10503 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.177651 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:14.172021 10089 catalog_manager.cc:5484] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20240507 08:05:14.180132 10155 leader_election.cc:341] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.152937 10306 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.184540 10407 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.188357 10499 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.188548 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:14.186089 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44638.8us
I20240507 08:05:14.191844 10338 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.194671 10383 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.195258 10383 ts_tablet_manager.cc:1367] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.331s user 0.013s sys 0.018s
I20240507 08:05:14.203503 10383 raft_consensus.cc:348] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.204528 10383 raft_consensus.cc:374] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.204880 10382 tablet_bootstrap.cc:654] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.205802 10338 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:14.191548 10503 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.206986 10156 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.189651 10407 raft_consensus.cc:686] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:14.208730 10503 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.209208 10407 consensus_queue.cc:237] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.209645 10503 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:14.210888 10503 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.212368 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.204876 10383 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.215122 10383 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.225348 10383 ts_tablet_manager.cc:1398] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.030s user 0.008s sys 0.002s
I20240507 08:05:14.219588 10469 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.229790 10383 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.239066 10089 catalog_manager.cc:5484] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.249657 10383 tablet_bootstrap.cc:654] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.252951 10381 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.253616 10381 ts_tablet_manager.cc:1367] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.180s user 0.023s sys 0.035s
I20240507 08:05:14.256392 10381 raft_consensus.cc:348] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.260283 10381 raft_consensus.cc:374] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.260547 10381 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.261162 10381 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.263444 10381 ts_tablet_manager.cc:1398] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240507 08:05:14.263676 10469 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.264349 10381 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.264201 10469 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.270499 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.270756 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.271399 10469 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.271940 10381 tablet_bootstrap.cc:654] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.272143 10264 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:14.274158 10382 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.273659 10306 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.274636 10382 ts_tablet_manager.cc:1367] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.176s user 0.017s sys 0.007s
I20240507 08:05:14.274966 10407 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.277017 10382 raft_consensus.cc:348] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.277535 10383 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.277737 10382 raft_consensus.cc:374] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.278781 10382 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.278945 10383 ts_tablet_manager.cc:1367] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.015s
I20240507 08:05:14.278084 10088 catalog_manager.cc:5484] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:14.281461 10383 raft_consensus.cc:348] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.282182 10383 raft_consensus.cc:374] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.282845 10383 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.283793 10383 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.284245 10382 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.286937 10383 ts_tablet_manager.cc:1398] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240507 08:05:14.284762 10407 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.287710 10381 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.288092 10383 tablet_bootstrap.cc:492] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.288127 10381 ts_tablet_manager.cc:1367] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.002s
I20240507 08:05:14.290920 10381 raft_consensus.cc:348] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.291639 10381 raft_consensus.cc:374] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.291891 10381 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.292531 10381 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.294596 10383 tablet_bootstrap.cc:654] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.290659 10087 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.296373 10382 ts_tablet_manager.cc:1398] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20240507 08:05:14.297273 10382 tablet_bootstrap.cc:492] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.302363 10383 tablet_bootstrap.cc:492] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.302870 10383 ts_tablet_manager.cc:1367] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20240507 08:05:14.303027 10382 tablet_bootstrap.cc:654] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.294631 10381 ts_tablet_manager.cc:1398] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:14.305397 10383 raft_consensus.cc:348] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.305286 10088 catalog_manager.cc:5484] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:14.306172 10381 tablet_bootstrap.cc:492] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.306255 10383 raft_consensus.cc:374] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.306680 10383 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.307307 10383 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.309393 10383 ts_tablet_manager.cc:1398] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:05:14.310331 10383 tablet_bootstrap.cc:492] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.312716 10381 tablet_bootstrap.cc:654] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.315505 10383 tablet_bootstrap.cc:654] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.316965 10382 tablet_bootstrap.cc:492] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.317415 10382 ts_tablet_manager.cc:1367] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240507 08:05:14.318538 10090 catalog_manager.cc:5484] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.319753 10382 raft_consensus.cc:348] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.320389 10382 raft_consensus.cc:374] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.320648 10382 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.321401 10382 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.327247 10382 ts_tablet_manager.cc:1398] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240507 08:05:14.328143 10382 tablet_bootstrap.cc:492] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.328325 10503 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.328920 10503 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.332679 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.333508 10338 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:14.334611 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.334983 10156 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:14.336009 10155 leader_election.cc:341] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.336232 10499 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.336709 10499 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.337122 10499 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.344352 10382 tablet_bootstrap.cc:654] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.345697 10503 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.347200 10499 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.356498 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:14.358233 10155 leader_election.cc:341] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.347625 10503 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.359359 10503 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.362699 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:14.366753 10155 leader_election.cc:341] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.368979 10503 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.351693 10381 tablet_bootstrap.cc:492] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.370071 10381 ts_tablet_manager.cc:1367] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.011s
I20240507 08:05:14.370249 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.370932 10338 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:14.371716 10383 tablet_bootstrap.cc:492] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.372089 10156 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.372289 10383 ts_tablet_manager.cc:1367] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.062s user 0.021s sys 0.034s
I20240507 08:05:14.372112 10499 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.373123 10499 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.373452 10499 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.373800 10499 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.374264 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.374873 10338 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.375214 10383 raft_consensus.cc:348] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.375878 10383 raft_consensus.cc:374] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.376150 10383 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.376739 10383 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.378768 10383 ts_tablet_manager.cc:1398] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20240507 08:05:14.380036 10383 tablet_bootstrap.cc:492] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.381593 10338 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:14.382705 10156 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.384347 10381 raft_consensus.cc:348] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.384924 10381 raft_consensus.cc:374] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.385134 10381 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.385855 10503 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.385785 10381 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.386387 10503 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:14.387269 10503 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.387867 10381 ts_tablet_manager.cc:1398] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20240507 08:05:14.390668 10383 tablet_bootstrap.cc:654] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.395828 10381 tablet_bootstrap.cc:492] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.399358 10382 tablet_bootstrap.cc:492] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.399848 10382 ts_tablet_manager.cc:1367] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.072s user 0.025s sys 0.025s
I20240507 08:05:14.404124 10382 raft_consensus.cc:348] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.405117 10382 raft_consensus.cc:374] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.405490 10382 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.406304 10382 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.409307 10381 tablet_bootstrap.cc:654] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.410605 10499 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.413641 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:14.415416 10155 leader_election.cc:341] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.419538 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.420131 10338 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.420341 10341 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.424448 10382 ts_tablet_manager.cc:1398] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20240507 08:05:14.426002 10483 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.428356 10483 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.431165 10341 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:14.432685 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.433672 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:14.434437 10231 leader_election.cc:341] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:14.435487 10232 leader_election.cc:341] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.435606 10156 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.436113 10232 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:14.436856 10426 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.437206 10435 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.437507 10426 raft_consensus.cc:686] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:14.438258 10483 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.439432 10426 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.440270 10382 tablet_bootstrap.cc:492] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.444576 10499 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.454450 10382 tablet_bootstrap.cc:654] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.455909 10087 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.464171 10382 tablet_bootstrap.cc:492] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.464771 10382 ts_tablet_manager.cc:1367] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20240507 08:05:14.467917 10382 raft_consensus.cc:348] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.468818 10382 raft_consensus.cc:374] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.469146 10382 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.469960 10382 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.473196 10382 ts_tablet_manager.cc:1398] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240507 08:05:14.476271 10499 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.480178 10382 tablet_bootstrap.cc:492] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.483249 10090 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.484858 10499 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.486814 10499 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.487181 10503 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.487706 10503 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.488780 10499 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.489535 10503 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.490593 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.490593 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.492527 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:14.493275 10155 leader_election.cc:341] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:14.493464 10156 leader_election.cc:341] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:14.494272 10155 leader_election.cc:341] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.494632 10155 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.495438 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.495635 10503 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.496058 10338 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:14.496948 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.497555 10503 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.497910 10503 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.498265 10503 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.505874 10382 tablet_bootstrap.cc:654] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.507716 10383 tablet_bootstrap.cc:492] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.507872 10381 tablet_bootstrap.cc:492] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.508211 10383 ts_tablet_manager.cc:1367] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.128s user 0.020s sys 0.068s
I20240507 08:05:14.508370 10381 ts_tablet_manager.cc:1367] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.113s user 0.027s sys 0.019s
I20240507 08:05:14.510833 10383 raft_consensus.cc:348] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.511487 10383 raft_consensus.cc:374] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.511736 10383 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.512342 10383 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.512496 10503 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.514360 10383 ts_tablet_manager.cc:1398] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240507 08:05:14.515316 10381 raft_consensus.cc:348] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.515969 10381 raft_consensus.cc:374] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.516217 10381 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.516409 10503 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.516830 10381 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.518473 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.518854 10381 ts_tablet_manager.cc:1398] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240507 08:05:14.519097 10383 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.519064 10338 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.519691 10381 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.525522 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:14.525946 10338 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:14.526302 10503 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:05:14.526954 10155 leader_election.cc:341] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.527748 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.528878 10426 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.529537 10426 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:14.530231 10381 tablet_bootstrap.cc:654] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.530632 10483 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.530489 10426 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.534606 10383 tablet_bootstrap.cc:654] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.531083 10483 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.535357 10382 tablet_bootstrap.cc:492] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.535813 10382 ts_tablet_manager.cc:1367] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.010s
I20240507 08:05:14.536787 10483 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.538259 10382 raft_consensus.cc:348] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.539072 10382 raft_consensus.cc:374] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.539341 10382 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.539448 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.539598 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.540498 10338 raft_consensus.cc:2379] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:14.539999 10382 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.543026 10382 ts_tablet_manager.cc:1398] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20240507 08:05:14.543967 10382 tablet_bootstrap.cc:492] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.548063 10232 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:14.549669 10483 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.550381 10382 tablet_bootstrap.cc:654] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.557845 10483 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.568899 10088 catalog_manager.cc:5484] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.570153 10382 tablet_bootstrap.cc:492] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.570592 10382 ts_tablet_manager.cc:1367] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.000s
I20240507 08:05:14.572993 10382 raft_consensus.cc:348] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.582720 10499 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.583207 10382 raft_consensus.cc:374] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.583513 10382 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.582172 10483 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.577358 10503 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.584493 10503 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.585788 10383 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.586230 10381 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.586266 10383 ts_tablet_manager.cc:1367] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.017s
I20240507 08:05:14.586454 10503 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.587112 10382 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.587579 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.588968 10383 raft_consensus.cc:348] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.589128 10382 ts_tablet_manager.cc:1398] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240507 08:05:14.589246 10503 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.589864 10383 raft_consensus.cc:374] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.590294 10383 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.590188 10503 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.592016 10382 tablet_bootstrap.cc:492] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
W20240507 08:05:14.589141 10156 leader_election.cc:341] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.593876 10503 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.586700 10381 ts_tablet_manager.cc:1367] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.016s
I20240507 08:05:14.591013 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.594543 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.595170 10338 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:14.591135 10383 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.596295 10156 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:14.597920 10382 tablet_bootstrap.cc:654] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.597076 10503 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.598994 10503 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.599447 10503 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.600219 10435 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.598260 10383 ts_tablet_manager.cc:1398] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240507 08:05:14.607635 10383 tablet_bootstrap.cc:492] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.608913 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.598392 10381 raft_consensus.cc:348] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.609720 10265 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:14.609772 10381 raft_consensus.cc:374] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.610535 10381 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
W20240507 08:05:14.611548 10155 leader_election.cc:341] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.611384 10381 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.612156 10155 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.614838 10381 ts_tablet_manager.cc:1398] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.020s user 0.004s sys 0.003s
I20240507 08:05:14.623921 10381 tablet_bootstrap.cc:492] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.630900 10381 tablet_bootstrap.cc:654] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.617054 10499 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.634486 10484 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.622552 10503 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.637632 10503 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.631528 10383 tablet_bootstrap.cc:654] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.616379 10382 tablet_bootstrap.cc:492] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.638428 10382 ts_tablet_manager.cc:1367] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.006s
I20240507 08:05:14.640980 10382 raft_consensus.cc:348] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.641912 10382 raft_consensus.cc:374] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.641960 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.642402 10382 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.642633 10341 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.643213 10382 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.652778 10483 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:14.659399 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:14.660027 10264 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.660446 10382 ts_tablet_manager.cc:1398] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.022s user 0.006s sys 0.002s
I20240507 08:05:14.662043 10382 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.665017 10341 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:14.666857 10488 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.673779 10426 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:14.685227 10488 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:14.687877 10156 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:14.693994 10426 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.694512 10426 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:14.695112 10488 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:14.695326 10426 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.701175 10383 tablet_bootstrap.cc:492] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.701709 10383 ts_tablet_manager.cc:1367] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.094s user 0.013s sys 0.004s
I20240507 08:05:14.704123 10383 raft_consensus.cc:348] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.704788 10383 raft_consensus.cc:374] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.705057 10383 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.707180 10469 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:14.706333 10383 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.724129 10381 tablet_bootstrap.cc:492] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.724725 10381 ts_tablet_manager.cc:1367] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.101s user 0.037s sys 0.042s
I20240507 08:05:14.726133 10264 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:14.727545 10381 raft_consensus.cc:348] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.728360 10381 raft_consensus.cc:374] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.728675 10381 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.729343 10381 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.733747 10383 ts_tablet_manager.cc:1398] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20240507 08:05:14.734596 10383 tablet_bootstrap.cc:492] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.731338 10469 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.735762 10381 ts_tablet_manager.cc:1398] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240507 08:05:14.737084 10381 tablet_bootstrap.cc:492] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.740336 10484 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.740950 10484 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.737756 10469 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.744186 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.744354 10469 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.744825 10188 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:14.745324 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.745798 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.746685 10232 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:14.747362 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.744186 10484 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.752316 10548 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.755525 10548 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.755971 10548 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.758589 10338 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
W20240507 08:05:14.759202 10306 leader_election.cc:341] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.760195 10265 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:14.761268 10304 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:14.769264 10407 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.786131 10382 tablet_bootstrap.cc:654] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.784305 10407 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.787624 10407 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.790597 10383 tablet_bootstrap.cc:654] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:14.803092 10155 consensus_peers.cc:485] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:14.804057 10436 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.806362 10548 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.807453 10407 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.809415 10407 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.810084 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:14.810634 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:14.811244 10262 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.811129 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:14.811394 10306 leader_election.cc:341] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.811827 10188 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.818588 10262 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:14.818287 10088 catalog_manager.cc:5484] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.819653 10304 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:14.809319 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.820782 10188 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:14.820943 10335 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.821229 10488 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.822535 10232 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:14.822332 10488 raft_consensus.cc:686] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:14.825661 10488 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.827883 10381 tablet_bootstrap.cc:654] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.825027 10484 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.833896 10484 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:14.834774 10484 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.821974 10548 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:14.839190 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:14.857362 10382 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.857822 10382 ts_tablet_manager.cc:1367] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.196s user 0.012s sys 0.027s
I20240507 08:05:14.861003 10090 catalog_manager.cc:5484] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:14.868675 10436 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.878026 10436 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.878501 10436 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.879474 10335 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:14.879069 10087 catalog_manager.cc:5484] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:14.880506 10436 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.885486 10381 tablet_bootstrap.cc:492] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:14.885983 10381 ts_tablet_manager.cc:1367] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.005s
I20240507 08:05:14.887537 10484 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.887995 10484 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.888702 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.889672 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.889909 10484 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:14.889968 10155 leader_election.cc:341] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.892428 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.892692 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:14.893970 10231 leader_election.cc:341] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:14.896298 10156 leader_election.cc:341] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.896505 10381 raft_consensus.cc:348] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.897218 10381 raft_consensus.cc:374] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.896747 10156 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:14.897485 10381 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.899165 10381 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.899276 10382 raft_consensus.cc:348] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.899914 10382 raft_consensus.cc:374] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.900135 10382 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.897853 10387 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:14.900758 10382 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.902279 10188 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:14.902518 10383 tablet_bootstrap.cc:492] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:14.902755 10382 ts_tablet_manager.cc:1398] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20240507 08:05:14.902961 10383 ts_tablet_manager.cc:1367] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.169s user 0.012s sys 0.009s
I20240507 08:05:14.903564 10382 tablet_bootstrap.cc:492] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.903405 10232 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:14.904407 10484 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.904736 10484 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.905046 10484 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.906462 10381 ts_tablet_manager.cc:1398] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.020s user 0.004s sys 0.001s
I20240507 08:05:14.907306 10381 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:14.905364 10383 raft_consensus.cc:348] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.908337 10383 raft_consensus.cc:374] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.908598 10383 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.911839 10484 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.912014 10383 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.919893 10382 tablet_bootstrap.cc:654] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.914500 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:14.920369 10188 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.920683 10381 tablet_bootstrap.cc:654] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.918167 10383 ts_tablet_manager.cc:1398] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240507 08:05:14.922583 10383 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:14.924448 10387 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:14.928165 10484 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.930047 10188 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:14.930860 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.931069 10382 tablet_bootstrap.cc:492] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.931129 10232 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:14.931619 10382 ts_tablet_manager.cc:1367] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.007s
I20240507 08:05:14.932029 10484 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:14.932507 10484 raft_consensus.cc:686] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:14.933300 10484 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.934224 10382 raft_consensus.cc:348] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:14.934846 10231 leader_election.cc:341] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:14.935026 10382 raft_consensus.cc:374] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.935338 10382 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.935854 10382 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.938282 10382 ts_tablet_manager.cc:1398] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20240507 08:05:14.939122 10382 tablet_bootstrap.cc:492] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.944341 10382 tablet_bootstrap.cc:654] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.955183 10382 tablet_bootstrap.cc:492] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.955631 10382 ts_tablet_manager.cc:1367] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20240507 08:05:14.958000 10382 raft_consensus.cc:348] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.958698 10382 raft_consensus.cc:374] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.958958 10382 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.959738 10382 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.961838 10382 ts_tablet_manager.cc:1398] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20240507 08:05:14.962307 10548 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.962636 10382 tablet_bootstrap.cc:492] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.962769 10548 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.964207 10488 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:14.964700 10548 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.964605 10488 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.966552 10488 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:14.967319 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.967957 10188 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:14.968562 10382 tablet_bootstrap.cc:654] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:14.969094 10232 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:14.970646 10548 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.971009 10548 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.971333 10548 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.974287 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:14.974923 10188 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:14.975384 10265 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.976037 10306 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:14.977622 10488 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:14.978003 10488 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:14.977842 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:14.978381 10488 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.979869 10382 tablet_bootstrap.cc:492] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:14.980333 10382 ts_tablet_manager.cc:1367] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20240507 08:05:14.981928 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:14.982168 10548 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.982718 10382 raft_consensus.cc:348] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.983364 10382 raft_consensus.cc:374] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:14.983631 10382 raft_consensus.cc:729] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:14.984230 10382 consensus_queue.cc:260] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:14.985417 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:14.986119 10264 raft_consensus.cc:2379] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:14.986824 10382 ts_tablet_manager.cc:1398] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:14.984577 10548 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:14.987202 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:14.987772 10188 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:14.989512 10382 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:14.993016 10499 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:05:15.011245 10155 consensus_peers.cc:485] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:15.013681 10499 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.017026 10488 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.019855 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.021167 10382 tablet_bootstrap.cc:654] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.021690 10341 raft_consensus.cc:2379] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:15.024752 10488 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.017046 10499 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20240507 08:05:15.028503 10383 tablet_bootstrap.cc:654] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.029383 10188 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:15.026072 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:15.030547 10232 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:15.031023 10265 raft_consensus.cc:2379] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:15.031672 10548 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.032223 10548 raft_consensus.cc:686] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:15.032071 10304 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.033047 10548 consensus_queue.cc:237] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.033501 10407 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.036661 10382 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.037115 10382 ts_tablet_manager.cc:1367] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.000s
I20240507 08:05:15.042937 10382 raft_consensus.cc:348] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.043665 10382 raft_consensus.cc:374] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.043964 10382 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.044970 10382 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.043292 10088 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:15.047047 10382 ts_tablet_manager.cc:1398] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20240507 08:05:15.053184 10499 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.053726 10499 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.054195 10382 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.048004 10381 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.056408 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.056599 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.057067 10265 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:15.057197 10341 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:15.058260 10156 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.059008 10436 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.059321 10436 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.059620 10436 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.065325 10382 tablet_bootstrap.cc:654] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.068837 10383 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.069330 10383 ts_tablet_manager.cc:1367] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.147s user 0.010s sys 0.005s
I20240507 08:05:15.065677 10499 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.071825 10383 raft_consensus.cc:348] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.072774 10383 raft_consensus.cc:374] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.073087 10383 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.066444 10436 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.073786 10383 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.076843 10436 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.075976 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:15.076269 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:15.077834 10335 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.077848 10264 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.056622 10381 ts_tablet_manager.cc:1367] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.149s user 0.009s sys 0.012s
I20240507 08:05:15.079428 10087 catalog_manager.cc:5484] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:15.081730 10436 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.081769 10381 raft_consensus.cc:348] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.082700 10381 raft_consensus.cc:374] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.082978 10381 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.083726 10381 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.085763 10383 ts_tablet_manager.cc:1398] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20240507 08:05:15.086016 10381 ts_tablet_manager.cc:1398] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:15.083057 10436 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.086685 10383 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.086757 10381 tablet_bootstrap.cc:492] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.089100 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:15.090463 10155 leader_election.cc:341] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.092134 10381 tablet_bootstrap.cc:654] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.095202 10335 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:15.096256 10436 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.096307 10156 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.097752 10264 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:15.097950 10436 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.098472 10436 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:15.099263 10436 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.099612 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:15.100836 10156 leader_election.cc:341] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.101269 10156 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.102175 10499 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.104660 10382 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.105017 10382 ts_tablet_manager.cc:1367] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.004s
I20240507 08:05:15.107242 10382 raft_consensus.cc:348] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.107931 10382 raft_consensus.cc:374] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.108197 10382 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.108855 10382 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.111263 10382 ts_tablet_manager.cc:1398] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240507 08:05:15.112164 10382 tablet_bootstrap.cc:492] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.113258 10383 tablet_bootstrap.cc:654] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.113276 10381 tablet_bootstrap.cc:492] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.115320 10381 ts_tablet_manager.cc:1367] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.005s
I20240507 08:05:15.118265 10382 tablet_bootstrap.cc:654] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.128913 10381 raft_consensus.cc:348] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.129671 10381 raft_consensus.cc:374] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.129976 10381 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.129743 10090 catalog_manager.cc:5484] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:15.130563 10381 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.135771 10381 ts_tablet_manager.cc:1398] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240507 08:05:15.136749 10381 tablet_bootstrap.cc:492] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.138414 10382 tablet_bootstrap.cc:492] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.138906 10382 ts_tablet_manager.cc:1367] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.013s
I20240507 08:05:15.141258 10382 raft_consensus.cc:348] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.142007 10382 raft_consensus.cc:374] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.142262 10382 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.142828 10382 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.144718 10381 tablet_bootstrap.cc:654] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.144716 10382 ts_tablet_manager.cc:1398] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:05:15.145824 10382 tablet_bootstrap.cc:492] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.152768 10382 tablet_bootstrap.cc:654] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.154631 10381 tablet_bootstrap.cc:492] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.154860 10383 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.155073 10381 ts_tablet_manager.cc:1367] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20240507 08:05:15.155267 10383 ts_tablet_manager.cc:1367] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.010s
I20240507 08:05:15.157465 10381 raft_consensus.cc:348] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.158224 10381 raft_consensus.cc:374] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.158474 10381 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.159498 10381 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.159603 10383 raft_consensus.cc:348] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.160303 10488 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.160306 10383 raft_consensus.cc:374] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.160881 10383 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.160899 10488 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.161425 10383 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.163633 10383 ts_tablet_manager.cc:1398] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240507 08:05:15.164832 10383 tablet_bootstrap.cc:492] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.165773 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.166411 10188 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:15.169952 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.170293 10383 tablet_bootstrap.cc:654] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.170534 10503 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.173700 10488 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.169524 10306 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:15.175707 10304 leader_election.cc:341] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.176059 10515 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.176555 10515 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.176975 10515 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.167229 10382 tablet_bootstrap.cc:492] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.178241 10382 ts_tablet_manager.cc:1367] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.000s
I20240507 08:05:15.181488 10382 raft_consensus.cc:348] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.182549 10382 raft_consensus.cc:374] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.182811 10382 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.183395 10382 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.185809 10381 ts_tablet_manager.cc:1398] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.030s user 0.010s sys 0.000s
I20240507 08:05:15.186856 10381 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.192075 10381 tablet_bootstrap.cc:654] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.195601 10382 ts_tablet_manager.cc:1398] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20240507 08:05:15.199501 10264 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.201719 10382 tablet_bootstrap.cc:492] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.202884 10381 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.203480 10381 ts_tablet_manager.cc:1367] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20240507 08:05:15.207597 10382 tablet_bootstrap.cc:654] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.220363 10381 raft_consensus.cc:348] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.221491 10381 raft_consensus.cc:374] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.221952 10381 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.223495 10381 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.226349 10381 ts_tablet_manager.cc:1398] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s
I20240507 08:05:15.227717 10381 tablet_bootstrap.cc:492] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.233952 10381 tablet_bootstrap.cc:654] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.232456 10382 tablet_bootstrap.cc:492] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.234680 10382 ts_tablet_manager.cc:1367] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.028s
I20240507 08:05:15.234104 10515 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.238021 10382 raft_consensus.cc:348] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.238734 10382 raft_consensus.cc:374] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.239042 10382 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.239851 10382 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.243372 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.243944 10188 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.248782 10382 ts_tablet_manager.cc:1398] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240507 08:05:15.249715 10382 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.250994 10387 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.254845 10382 tablet_bootstrap.cc:654] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.256318 10426 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.256856 10426 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.259223 10426 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.259557 10188 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:15.260919 10306 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.261859 10488 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.262370 10488 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:15.262563 10383 tablet_bootstrap.cc:492] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.263376 10383 ts_tablet_manager.cc:1367] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.099s user 0.033s sys 0.037s
I20240507 08:05:15.263362 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.263249 10488 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.267006 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.267127 10383 raft_consensus.cc:348] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.271520 10383 raft_consensus.cc:374] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.272367 10383 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.274096 10383 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:15.276595 10156 leader_election.cc:341] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.278344 10383 ts_tablet_manager.cc:1398] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
W20240507 08:05:15.278867 10155 leader_election.cc:341] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.279523 10155 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.279992 10383 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.281745 10436 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.283751 10381 tablet_bootstrap.cc:492] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.284229 10381 ts_tablet_manager.cc:1367] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.057s user 0.007s sys 0.006s
I20240507 08:05:15.286934 10383 tablet_bootstrap.cc:654] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.286760 10381 raft_consensus.cc:348] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.287429 10381 raft_consensus.cc:374] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.287716 10381 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.288390 10381 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.292457 10381 ts_tablet_manager.cc:1398] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240507 08:05:15.293187 10484 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.293596 10381 tablet_bootstrap.cc:492] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.293673 10484 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.295868 10484 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.296356 10382 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.296813 10382 ts_tablet_manager.cc:1367] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.016s
I20240507 08:05:15.297449 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.298135 10188 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:15.298988 10383 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.299489 10383 ts_tablet_manager.cc:1367] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20240507 08:05:15.299422 10232 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:15.299942 10086 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:15.302131 10383 raft_consensus.cc:348] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.303043 10383 raft_consensus.cc:374] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.303354 10383 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.304081 10383 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.299225 10382 raft_consensus.cc:348] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.305068 10382 raft_consensus.cc:374] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.305250 10381 tablet_bootstrap.cc:654] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.305404 10382 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.306104 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:15.306797 10435 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.307163 10435 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.307497 10435 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.307958 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.308542 10335 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:15.259209 10515 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:15.314545 10304 leader_election.cc:341] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.316808 10435 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.321612 10435 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.306207 10382 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.322376 10381 tablet_bootstrap.cc:492] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.322803 10381 ts_tablet_manager.cc:1367] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.000s
I20240507 08:05:15.323416 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.319625 10383 ts_tablet_manager.cc:1398] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20240507 08:05:15.324023 10188 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.324702 10383 tablet_bootstrap.cc:492] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.325137 10381 raft_consensus.cc:348] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.324970 10382 ts_tablet_manager.cc:1398] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
I20240507 08:05:15.325824 10381 raft_consensus.cc:374] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.326120 10381 raft_consensus.cc:729] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.326591 10382 tablet_bootstrap.cc:492] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.330335 10383 tablet_bootstrap.cc:654] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.331153 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:15.331727 10335 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.336144 10382 tablet_bootstrap.cc:654] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.326836 10381 consensus_queue.cc:260] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.339191 10381 ts_tablet_manager.cc:1398] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20240507 08:05:15.340376 10381 tablet_bootstrap.cc:492] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.340662 10188 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:15.340955 10186 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.345065 10232 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:15.346045 10435 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.346518 10435 raft_consensus.cc:686] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:15.347441 10515 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.347301 10435 consensus_queue.cc:237] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.355389 10381 tablet_bootstrap.cc:654] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.356784 10436 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.357467 10436 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.361133 10426 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.370668 10335 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:15.375772 10382 tablet_bootstrap.cc:492] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.376327 10382 ts_tablet_manager.cc:1367] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.050s user 0.015s sys 0.002s
I20240507 08:05:15.376610 10515 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.377040 10436 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.376314 10086 catalog_manager.cc:5484] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:15.367492 10426 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.379805 10426 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.380695 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.381515 10484 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.382161 10484 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:15.387368 10156 leader_election.cc:341] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.387403 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.388762 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.390815 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.433074 10382 raft_consensus.cc:348] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.433629 10382 raft_consensus.cc:374] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.433830 10382 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.409301 10383 tablet_bootstrap.cc:492] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.434867 10383 ts_tablet_manager.cc:1367] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.110s user 0.018s sys 0.027s
I20240507 08:05:15.408769 10387 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.435315 10382 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.435904 10387 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.437858 10387 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.390208 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.438270 10383 raft_consensus.cc:348] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.439039 10383 raft_consensus.cc:374] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.439350 10383 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.438930 10188 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:15.389416 10484 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.442255 10232 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:15.443006 10382 ts_tablet_manager.cc:1398] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20240507 08:05:15.443740 10382 tablet_bootstrap.cc:492] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.440339 10383 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.394598 10436 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.446192 10515 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.446647 10515 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.445914 10436 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.413275 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.432305 10336 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:15.384617 10435 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.056s
W20240507 08:05:15.451205 10231 leader_election.cc:341] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.452543 10515 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.454077 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.458530 10383 ts_tablet_manager.cc:1398] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
W20240507 08:05:15.460247 10304 leader_election.cc:341] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.461738 10383 tablet_bootstrap.cc:492] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.450382 10484 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.454077 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.467578 10381 tablet_bootstrap.cc:492] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.467921 10381 ts_tablet_manager.cc:1367] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.128s user 0.015s sys 0.001s
I20240507 08:05:15.465744 10156 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.470597 10387 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.470947 10387 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.471241 10189 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:15.471246 10387 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.476143 10306 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.478976 10436 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.481027 10387 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.482123 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.482865 10387 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:15.483649 10156 leader_election.cc:341] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.473979 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.484236 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:15.484804 10336 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:15.485186 10156 leader_election.cc:341] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.466255 10484 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.487829 10381 raft_consensus.cc:348] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.488312 10381 raft_consensus.cc:374] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.488608 10381 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.489200 10381 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.490929 10515 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.492341 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.493247 10515 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.493674 10515 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.497308 10548 consensus_queue.cc:1035] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.498999 10381 ts_tablet_manager.cc:1398] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.031s user 0.004s sys 0.001s
I20240507 08:05:15.500463 10381 tablet_bootstrap.cc:492] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.500926 10338 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.488174 10484 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.505728 10336 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:15.512646 10515 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.518942 10515 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.513772 10435 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240507 08:05:15.520618 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.521291 10186 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.526747 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:15.533747 10304 leader_election.cc:341] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.542910 10186 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:15.544430 10306 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:15.523619 10155 leader_election.cc:341] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.545374 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.546380 10515 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.548413 10515 raft_consensus.cc:686] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:15.557013 10382 tablet_bootstrap.cc:654] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.550807 10156 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.550577 10387 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.559710 10436 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:05:15.562129 10155 leader_election.cc:341] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.563421 10383 tablet_bootstrap.cc:654] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:15.554060 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 57372us
I20240507 08:05:15.566293 10436 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:15.574771 10436 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.575881 10381 tablet_bootstrap.cc:654] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.567122 10515 consensus_queue.cc:237] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.582355 10387 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.596374 10387 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.596889 10387 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.602300 10387 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.593227 10548 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20240507 08:05:15.623659 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.628042 10484 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:15.632154 10156 leader_election.cc:341] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.633955 10265 raft_consensus.cc:2379] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:15.637610 10435 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:15.645028 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:15.646248 10089 catalog_manager.cc:5484] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:15.647747 10484 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.657625 10382 tablet_bootstrap.cc:492] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
W20240507 08:05:15.655826 10231 leader_election.cc:341] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.660743 10382 ts_tablet_manager.cc:1367] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.217s user 0.021s sys 0.009s
I20240507 08:05:15.669466 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.670596 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.671206 10188 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.675208 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:15.677911 10548 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.678364 10548 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.678716 10383 tablet_bootstrap.cc:492] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.679409 10383 ts_tablet_manager.cc:1367] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.220s user 0.027s sys 0.015s
I20240507 08:05:15.681118 10603 consensus_queue.cc:1035] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.682965 10407 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.683435 10407 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.689502 10548 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.689695 10436 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.697069 10383 raft_consensus.cc:348] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.697845 10383 raft_consensus.cc:374] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.698066 10383 raft_consensus.cc:729] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.699266 10383 consensus_queue.cc:260] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.701880 10264 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.703323 10382 raft_consensus.cc:348] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.703999 10382 raft_consensus.cc:374] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.704289 10382 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.704599 10188 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:15.704957 10382 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.708396 10383 ts_tablet_manager.cc:1398] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20240507 08:05:15.709213 10383 tablet_bootstrap.cc:492] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.710994 10382 ts_tablet_manager.cc:1398] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20240507 08:05:15.711900 10382 tablet_bootstrap.cc:492] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.706398 10407 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.708320 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.717173 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.699257 10436 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.724434 10232 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:15.725337 10435 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.725817 10435 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:15.727823 10381 tablet_bootstrap.cc:492] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.728202 10381 ts_tablet_manager.cc:1367] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.228s user 0.023s sys 0.033s
I20240507 08:05:15.719792 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:15.735987 10306 leader_election.cc:341] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.726640 10435 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.738140 10436 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.738783 10407 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20240507 08:05:15.742961 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:15.743175 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:15.744777 10156 leader_election.cc:341] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.745039 10484 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.745193 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:15.747805 10304 leader_election.cc:341] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.748255 10304 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.734570 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.749457 10407 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.752630 10383 tablet_bootstrap.cc:654] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.757357 10155 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.757437 10407 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240507 08:05:15.758674 10155 leader_election.cc:341] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.759106 10155 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.759816 10499 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.760416 10436 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.760870 10265 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.769099 10381 raft_consensus.cc:348] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.769757 10381 raft_consensus.cc:374] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.770028 10381 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.770781 10381 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:15.776881 10155 leader_election.cc:341] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.779325 10499 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:05:15.778959 10155 leader_election.cc:341] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.780625 10382 tablet_bootstrap.cc:654] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.781898 10155 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.784233 10499 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.788523 10381 ts_tablet_manager.cc:1398] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.060s user 0.006s sys 0.000s
I20240507 08:05:15.789824 10381 tablet_bootstrap.cc:492] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.815110 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.832139 10381 tablet_bootstrap.cc:654] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:15.833815 10155 leader_election.cc:341] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.836848 10155 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.837476 10436 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.871573 10383 tablet_bootstrap.cc:492] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.872115 10383 ts_tablet_manager.cc:1367] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.163s user 0.013s sys 0.004s
I20240507 08:05:15.874512 10383 raft_consensus.cc:348] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.875185 10383 raft_consensus.cc:374] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.875413 10383 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.875870 10383 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.863584 10090 catalog_manager.cc:5484] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:15.881035 10383 ts_tablet_manager.cc:1398] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20240507 08:05:15.881907 10383 tablet_bootstrap.cc:492] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.882525 10382 tablet_bootstrap.cc:492] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.883029 10382 ts_tablet_manager.cc:1367] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.171s user 0.013s sys 0.006s
I20240507 08:05:15.883528 10515 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.884013 10515 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.886499 10515 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.887079 10383 tablet_bootstrap.cc:654] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.886539 10382 raft_consensus.cc:348] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.887516 10382 raft_consensus.cc:374] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.887861 10382 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.888361 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.888531 10382 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.888991 10264 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:15.886816 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.890709 10381 tablet_bootstrap.cc:492] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.898365 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:15.898739 10381 ts_tablet_manager.cc:1367] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.109s user 0.017s sys 0.004s
I20240507 08:05:15.899252 10515 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.899508 10515 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.899720 10515 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.900969 10381 raft_consensus.cc:348] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.901662 10381 raft_consensus.cc:374] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.901921 10381 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.902551 10186 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:15.905771 10383 tablet_bootstrap.cc:492] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.906176 10383 ts_tablet_manager.cc:1367] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.003s
I20240507 08:05:15.905565 10515 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.908596 10383 raft_consensus.cc:348] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.909266 10383 raft_consensus.cc:374] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.908759 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.909627 10383 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.909348 10435 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:15.910140 10515 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.909860 10381 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.910401 10383 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.911477 10382 ts_tablet_manager.cc:1398] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20240507 08:05:15.912014 10381 ts_tablet_manager.cc:1398] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20240507 08:05:15.912356 10382 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.912449 10383 ts_tablet_manager.cc:1398] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:15.912794 10381 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:15.913264 10383 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.913735 10436 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.914155 10436 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.916069 10436 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:15.917749 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.918108 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:15.918928 10383 tablet_bootstrap.cc:654] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:15.919054 10155 leader_election.cc:341] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:15.919483 10156 leader_election.cc:341] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:15.919919 10156 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:15.921273 10188 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.921530 10381 tablet_bootstrap.cc:654] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.923224 10086 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:15.909658 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:15.926255 10264 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:15.928891 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:15.932243 10382 tablet_bootstrap.cc:654] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.933773 10621 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:15.938416 10381 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:15.938930 10381 ts_tablet_manager.cc:1367] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.004s
I20240507 08:05:15.942242 10381 raft_consensus.cc:348] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.953331 10381 raft_consensus.cc:374] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.953637 10381 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.954221 10381 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.955050 10188 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:15.956140 10264 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:15.956233 10306 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.957098 10515 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:15.957535 10515 raft_consensus.cc:686] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:15.958119 10407 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.958346 10515 consensus_queue.cc:237] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.958606 10407 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.960374 10407 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:15.963507 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.964135 10188 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:15.965209 10306 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:15.966012 10488 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:15.966347 10488 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:15.966678 10488 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.968626 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:15.969276 10267 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:15.970145 10383 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:15.970690 10383 ts_tablet_manager.cc:1367] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.023s
I20240507 08:05:15.972208 10435 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:15.972710 10488 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.974020 10383 raft_consensus.cc:348] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.974726 10383 raft_consensus.cc:374] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.975014 10383 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.975158 10382 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:15.975548 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:15.975747 10382 ts_tablet_manager.cc:1367] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.003s
I20240507 08:05:15.976110 10188 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.975697 10383 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.978119 10383 ts_tablet_manager.cc:1398] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240507 08:05:15.978978 10383 tablet_bootstrap.cc:492] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:15.981040 10382 raft_consensus.cc:348] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.981776 10382 raft_consensus.cc:374] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:15.982071 10382 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:15.982671 10382 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.984433 10383 tablet_bootstrap.cc:654] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.984845 10382 ts_tablet_manager.cc:1398] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240507 08:05:15.987017 10382 tablet_bootstrap.cc:492] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:15.988356 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:15.988917 10267 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:15.993162 10382 tablet_bootstrap.cc:654] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:15.995146 10435 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:15.995765 10435 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:15.998345 10267 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:15.999212 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:15.999680 10304 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:16.000319 10189 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:16.000437 10407 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:16.001027 10407 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:16.002255 10407 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.003693 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.004916 10435 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.006035 10488 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.006129 10231 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:16.006975 10435 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.007763 10381 ts_tablet_manager.cc:1398] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.069s user 0.007s sys 0.000s
I20240507 08:05:16.008862 10381 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.010658 10382 tablet_bootstrap.cc:492] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.011164 10382 ts_tablet_manager.cc:1367] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20240507 08:05:16.011480 10188 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:16.013855 10382 raft_consensus.cc:348] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.016183 10382 raft_consensus.cc:374] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.016793 10382 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.018278 10382 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.021917 10382 ts_tablet_manager.cc:1398] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240507 08:05:16.022874 10382 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.047542 10382 tablet_bootstrap.cc:654] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.070302 10616 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.071014 10616 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.071883 10089 catalog_manager.cc:5484] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:16.073647 10616 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.074702 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.075342 10338 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:16.076390 10231 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:16.080559 10616 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.083195 10616 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.083554 10616 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.083698 10435 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:16.091388 10381 tablet_bootstrap.cc:654] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.091621 10616 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.092777 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.093418 10188 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:16.093979 10616 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.095117 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:16.095816 10383 tablet_bootstrap.cc:492] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.095778 10338 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.096350 10383 ts_tablet_manager.cc:1367] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.118s user 0.017s sys 0.051s
I20240507 08:05:16.096148 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:16.097678 10189 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.098771 10383 raft_consensus.cc:348] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.099440 10383 raft_consensus.cc:374] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.099689 10383 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.100309 10383 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.103196 10515 consensus_queue.cc:1035] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:16.111737 10381 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.117413 10381 ts_tablet_manager.cc:1367] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.004s
I20240507 08:05:16.120225 10189 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:16.107451 10338 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:16.120836 10381 raft_consensus.cc:348] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.121637 10381 raft_consensus.cc:374] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.121512 10232 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:16.122010 10381 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.129660 10616 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:16.131290 10616 raft_consensus.cc:686] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:16.115221 10086 catalog_manager.cc:5484] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:16.132400 10616 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.139289 10382 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.139724 10382 ts_tablet_manager.cc:1367] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.009s
I20240507 08:05:16.141747 10382 raft_consensus.cc:348] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.142400 10382 raft_consensus.cc:374] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.142668 10382 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.143245 10382 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.144711 10382 ts_tablet_manager.cc:1398] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240507 08:05:16.145551 10382 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.150079 10469 consensus_queue.cc:1035] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240507 08:05:16.157914 10381 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.158499 10383 ts_tablet_manager.cc:1398] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.062s user 0.006s sys 0.002s
I20240507 08:05:16.159497 10436 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:16.160025 10381 ts_tablet_manager.cc:1398] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.042s user 0.001s sys 0.006s
I20240507 08:05:16.160851 10381 tablet_bootstrap.cc:492] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.161976 10488 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.162441 10488 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.164307 10488 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.165354 10382 tablet_bootstrap.cc:654] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.170998 10381 tablet_bootstrap.cc:654] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.171916 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.172602 10189 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:16.174093 10306 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:16.174176 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.178668 10381 tablet_bootstrap.cc:492] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
W20240507 08:05:16.178810 10304 leader_election.cc:341] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.179134 10381 ts_tablet_manager.cc:1367] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
W20240507 08:05:16.179852 10155 consensus_peers.cc:485] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:16.180900 10469 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.181290 10469 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.181681 10469 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.183123 10383 tablet_bootstrap.cc:492] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.184123 10381 raft_consensus.cc:348] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.184826 10381 raft_consensus.cc:374] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.185081 10381 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.185742 10381 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.191395 10381 ts_tablet_manager.cc:1398] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.012s user 0.003s sys 0.003s
I20240507 08:05:16.192226 10381 tablet_bootstrap.cc:492] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.193347 10383 tablet_bootstrap.cc:654] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.198699 10335 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.196143 10381 tablet_bootstrap.cc:654] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.208204 10089 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:16.196305 10469 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.231353 10469 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.234131 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:16.234412 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:16.235037 10616 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:05:16.235673 10304 leader_election.cc:341] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.235072 10189 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.241629 10381 tablet_bootstrap.cc:492] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.242079 10381 ts_tablet_manager.cc:1367] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.001s
I20240507 08:05:16.248737 10606 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:16.249610 10189 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:16.250699 10382 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.251497 10306 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.251201 10382 ts_tablet_manager.cc:1367] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.106s user 0.019s sys 0.018s
I20240507 08:05:16.253819 10469 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:16.254308 10469 raft_consensus.cc:686] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:16.255132 10469 consensus_queue.cc:237] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.259346 10383 tablet_bootstrap.cc:492] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.259796 10383 ts_tablet_manager.cc:1367] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.077s user 0.018s sys 0.007s
I20240507 08:05:16.262806 10381 raft_consensus.cc:348] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.263597 10381 raft_consensus.cc:374] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.263391 10382 raft_consensus.cc:348] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.263864 10381 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.264088 10382 raft_consensus.cc:374] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.264358 10382 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.266525 10383 raft_consensus.cc:348] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.267489 10383 raft_consensus.cc:374] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.266837 10381 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.267753 10383 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.268373 10383 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.270521 10383 ts_tablet_manager.cc:1398] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.010s user 0.003s sys 0.003s
I20240507 08:05:16.271342 10383 tablet_bootstrap.cc:492] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.276084 10381 ts_tablet_manager.cc:1398] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.034s user 0.004s sys 0.001s
I20240507 08:05:16.276456 10383 tablet_bootstrap.cc:654] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.277048 10381 tablet_bootstrap.cc:492] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.265152 10382 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.280272 10382 ts_tablet_manager.cc:1398] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20240507 08:05:16.284849 10382 tablet_bootstrap.cc:492] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.285156 10488 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.285624 10488 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.287497 10488 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.290066 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.290767 10189 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:16.291569 10382 tablet_bootstrap.cc:654] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.292263 10306 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:16.293885 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.294536 10267 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:16.296355 10488 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.296756 10488 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.297122 10488 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.305379 10381 tablet_bootstrap.cc:654] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.313563 10499 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.314080 10499 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.314785 10090 catalog_manager.cc:5484] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:16.315981 10499 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.318374 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.318872 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.312804 10383 tablet_bootstrap.cc:492] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.319579 10383 ts_tablet_manager.cc:1367] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.004s
W20240507 08:05:16.319842 10155 leader_election.cc:341] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:16.320842 10156 leader_election.cc:341] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.321260 10156 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.322014 10620 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:16.321933 10383 raft_consensus.cc:348] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.322701 10383 raft_consensus.cc:374] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.322988 10383 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.322923 10488 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.323666 10383 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.325524 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:16.326174 10186 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.328047 10382 tablet_bootstrap.cc:492] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.328491 10382 ts_tablet_manager.cc:1367] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.012s
I20240507 08:05:16.330785 10382 raft_consensus.cc:348] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.331459 10382 raft_consensus.cc:374] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.331718 10382 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.332317 10382 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.334576 10383 ts_tablet_manager.cc:1398] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20240507 08:05:16.334697 10186 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:16.335788 10306 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:16.336238 10382 ts_tablet_manager.cc:1398] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20240507 08:05:16.336760 10383 tablet_bootstrap.cc:492] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.338889 10488 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.339112 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:16.339444 10488 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:16.339686 10267 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.340080 10488 raft_consensus.cc:686] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:16.340881 10488 consensus_queue.cc:237] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.346422 10267 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:16.347740 10382 tablet_bootstrap.cc:492] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.347852 10381 tablet_bootstrap.cc:492] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.348276 10381 ts_tablet_manager.cc:1367] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.000s
I20240507 08:05:16.350587 10381 raft_consensus.cc:348] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.351277 10381 raft_consensus.cc:374] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.352910 10383 tablet_bootstrap.cc:654] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.352973 10381 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.353753 10381 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.355995 10381 ts_tablet_manager.cc:1398] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240507 08:05:16.356798 10381 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.360445 10382 tablet_bootstrap.cc:654] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.365031 10086 catalog_manager.cc:5484] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:16.374946 10383 tablet_bootstrap.cc:492] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.375465 10383 ts_tablet_manager.cc:1367] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20240507 08:05:16.377902 10383 raft_consensus.cc:348] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.378607 10383 raft_consensus.cc:374] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.378872 10383 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.379549 10383 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.381736 10383 ts_tablet_manager.cc:1398] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20240507 08:05:16.383505 10381 tablet_bootstrap.cc:654] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.387017 10383 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.388597 10382 tablet_bootstrap.cc:492] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.389284 10382 ts_tablet_manager.cc:1367] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.006s
I20240507 08:05:16.393049 10383 tablet_bootstrap.cc:654] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.392719 10382 raft_consensus.cc:348] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.393947 10382 raft_consensus.cc:374] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.394385 10382 raft_consensus.cc:729] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.397648 10381 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.398101 10381 ts_tablet_manager.cc:1367] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.041s user 0.019s sys 0.000s
I20240507 08:05:16.400416 10383 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.400832 10383 ts_tablet_manager.cc:1367] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240507 08:05:16.400650 10382 consensus_queue.cc:260] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.403085 10382 ts_tablet_manager.cc:1398] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20240507 08:05:16.403124 10383 raft_consensus.cc:348] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.403966 10383 raft_consensus.cc:374] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.404086 10382 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.404263 10383 raft_consensus.cc:729] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.405117 10383 consensus_queue.cc:260] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.406927 10383 ts_tablet_manager.cc:1398] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240507 08:05:16.407745 10383 tablet_bootstrap.cc:492] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.412276 10383 tablet_bootstrap.cc:654] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.414584 10382 tablet_bootstrap.cc:654] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.401437 10381 raft_consensus.cc:348] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.416841 10381 raft_consensus.cc:374] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.417096 10381 raft_consensus.cc:729] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.417515 10381 consensus_queue.cc:260] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.419821 10381 ts_tablet_manager.cc:1398] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240507 08:05:16.423048 10383 tablet_bootstrap.cc:492] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.423503 10383 ts_tablet_manager.cc:1367] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240507 08:05:16.423846 10381 tablet_bootstrap.cc:492] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.425710 10383 raft_consensus.cc:348] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.426348 10383 raft_consensus.cc:374] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.426617 10383 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.427210 10383 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.428630 10469 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.429176 10469 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.430948 10382 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.431403 10382 ts_tablet_manager.cc:1367] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.007s
I20240507 08:05:16.431146 10469 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.432772 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.433305 10383 ts_tablet_manager.cc:1398] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20240507 08:05:16.434232 10383 tablet_bootstrap.cc:492] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.433779 10382 raft_consensus.cc:348] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.434782 10382 raft_consensus.cc:374] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.435155 10382 raft_consensus.cc:729] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.435829 10382 consensus_queue.cc:260] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.433398 10186 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:16.438122 10382 ts_tablet_manager.cc:1398] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240507 08:05:16.439167 10382 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.439280 10306 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:16.440230 10488 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.440665 10383 tablet_bootstrap.cc:654] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.440606 10488 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.441057 10488 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.443944 10499 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.444403 10499 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.445969 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.446377 10499 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.447839 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.448202 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.448061 10488 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:16.449185 10156 leader_election.cc:341] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:16.449658 10155 leader_election.cc:341] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.450086 10155 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.450788 10499 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:05:16.451220 10304 leader_election.cc:341] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.452255 10382 tablet_bootstrap.cc:654] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.453136 10488 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.454174 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:16.455626 10304 leader_election.cc:341] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.456732 10381 tablet_bootstrap.cc:654] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.458706 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:16.459309 10186 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.463896 10382 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.464352 10382 ts_tablet_manager.cc:1367] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.000s
I20240507 08:05:16.466778 10382 raft_consensus.cc:348] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.470463 10382 raft_consensus.cc:374] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.470737 10382 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.471340 10382 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.473768 10382 ts_tablet_manager.cc:1398] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20240507 08:05:16.474900 10382 tablet_bootstrap.cc:492] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.476477 10604 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.476878 10381 tablet_bootstrap.cc:492] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.477506 10381 ts_tablet_manager.cc:1367] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.007s
I20240507 08:05:16.476946 10604 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.479614 10604 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.479907 10381 raft_consensus.cc:348] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.480705 10381 raft_consensus.cc:374] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.480966 10381 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.481067 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.481731 10381 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.482251 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.482604 10382 tablet_bootstrap.cc:654] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.483852 10381 ts_tablet_manager.cc:1398] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:05:16.484102 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:16.485672 10606 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:16.487821 10186 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:16.491043 10306 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.492688 10469 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:16.493324 10469 raft_consensus.cc:686] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:16.494449 10469 consensus_queue.cc:237] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.499980 10383 tablet_bootstrap.cc:492] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.500443 10383 ts_tablet_manager.cc:1367] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.004s
I20240507 08:05:16.500833 10382 tablet_bootstrap.cc:492] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.501281 10382 ts_tablet_manager.cc:1367] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.010s
I20240507 08:05:16.501627 10499 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.502164 10499 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.502794 10383 raft_consensus.cc:348] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.503470 10383 raft_consensus.cc:374] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.503707 10383 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.504096 10499 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.504267 10383 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.504051 10382 raft_consensus.cc:348] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.506390 10382 raft_consensus.cc:374] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.506662 10382 raft_consensus.cc:729] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.507115 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.507239 10382 consensus_queue.cc:260] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.507772 10383 ts_tablet_manager.cc:1398] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20240507 08:05:16.508848 10381 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.509212 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.509632 10488 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.510169 10383 tablet_bootstrap.cc:492] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.510425 10382 ts_tablet_manager.cc:1398] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20240507 08:05:16.510052 10488 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.511898 10488 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.513298 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.513932 10186 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
W20240507 08:05:16.514823 10156 leader_election.cc:341] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.515087 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:16.516088 10155 leader_election.cc:341] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.516412 10488 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.516530 10155 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.516781 10488 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.517171 10488 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.517606 10621 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:16.519357 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:16.523718 10304 leader_election.cc:341] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.524679 10488 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.525528 10087 catalog_manager.cc:5484] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:16.527247 10381 tablet_bootstrap.cc:654] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.527316 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:16.527941 10186 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.534548 10383 tablet_bootstrap.cc:654] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.535111 10186 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:16.536157 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:16.537067 10515 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:16.537706 10382 tablet_bootstrap.cc:492] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.538357 10515 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:16.538806 10515 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:16.539553 10515 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.554864 10499 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.556373 10499 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.558853 10488 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.564947 10383 tablet_bootstrap.cc:492] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.565317 10383 ts_tablet_manager.cc:1367] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.006s
I20240507 08:05:16.569025 10499 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.570626 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:16.572368 10156 leader_election.cc:341] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.574903 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:16.576561 10304 leader_election.cc:341] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.581745 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.575954 10488 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240507 08:05:16.585934 10383 raft_consensus.cc:348] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.587242 10383 raft_consensus.cc:374] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.587532 10383 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.587903 10381 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.588337 10381 ts_tablet_manager.cc:1367] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.080s user 0.016s sys 0.013s
I20240507 08:05:16.589725 10383 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.589306 10087 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
W20240507 08:05:16.590968 10155 leader_election.cc:341] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.591533 10155 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.594555 10469 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:16.596577 10621 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:16.596889 10260 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.597716 10383 ts_tablet_manager.cc:1398] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.032s user 0.008s sys 0.000s
I20240507 08:05:16.600828 10383 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:16.605427 10382 tablet_bootstrap.cc:654] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.612612 10381 raft_consensus.cc:348] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.614476 10381 raft_consensus.cc:374] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.614866 10381 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.616340 10381 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.625247 10604 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.625823 10604 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.638705 10381 ts_tablet_manager.cc:1398] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20240507 08:05:16.639494 10381 tablet_bootstrap.cc:492] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:16.642272 10604 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.644963 10515 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:16.657519 10606 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.658226 10606 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.660658 10603 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.661293 10603 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.671409 10382 tablet_bootstrap.cc:492] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:16.671888 10382 ts_tablet_manager.cc:1367] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.134s user 0.017s sys 0.007s
I20240507 08:05:16.673426 10383 tablet_bootstrap.cc:654] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.672674 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.678762 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.678908 10189 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:16.679456 10341 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:16.688244 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.689848 10189 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:16.706037 10606 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.707353 10620 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.708258 10381 tablet_bootstrap.cc:654] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:16.707861 10620 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.694444 10603 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.715083 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:16.716482 10155 leader_election.cc:341] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.717166 10620 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.717866 10407 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.718430 10407 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.722563 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.723209 10189 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:16.720782 10407 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.734603 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:16.736994 10156 leader_election.cc:341] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:16.737638 10156 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:16.738144 10306 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:16.740319 10488 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.740809 10488 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.741212 10488 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.741513 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:16.750320 10189 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:16.764775 10620 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:16.774760 10488 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.889276 10488 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.889860 10382 raft_consensus.cc:348] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.890825 10382 raft_consensus.cc:374] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.891161 10382 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.889900 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.891955 10382 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.895535 10264 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:16.898200 10267 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.904460 10382 ts_tablet_manager.cc:1398] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.232s user 0.015s sys 0.025s
I20240507 08:05:16.904197 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.905388 10382 tablet_bootstrap.cc:492] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:16.907171 10381 tablet_bootstrap.cc:492] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:16.907599 10381 ts_tablet_manager.cc:1367] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.268s user 0.014s sys 0.012s
I20240507 08:05:16.908802 10383 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:16.909276 10383 ts_tablet_manager.cc:1367] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.309s user 0.012s sys 0.003s
I20240507 08:05:16.887967 10232 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:16.916440 10436 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.920753 10499 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.925765 10407 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.928084 10676 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.924258 10436 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.922482 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:16.932224 10499 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.933482 10232 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:16.942807 10675 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.943274 10675 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.943871 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:16.944535 10188 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.935346 10676 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:16.955191 10381 raft_consensus.cc:348] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.956135 10381 raft_consensus.cc:374] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.956472 10381 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.957278 10381 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.946208 10499 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.952864 10341 raft_consensus.cc:2379] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:16.939882 10407 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.951491 10616 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:16.959833 10436 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:16.953377 10676 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:16.971225 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:16.973583 10383 raft_consensus.cc:348] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.975138 10383 raft_consensus.cc:374] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:16.975493 10383 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:16.976235 10383 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:16.971735 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:16.967031 10616 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:16.992996 10678 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.085s user 0.002s sys 0.000s
I20240507 08:05:16.997045 10381 ts_tablet_manager.cc:1398] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.089s user 0.010s sys 0.011s
I20240507 08:05:16.986678 10407 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:16.967872 10604 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.979914 10603 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:16.996577 10616 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.006161 10603 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.007977 10383 ts_tablet_manager.cc:1398] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.098s user 0.007s sys 0.000s
I20240507 08:05:17.009171 10383 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:17.000438 10381 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:17.001530 10604 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.012064 10603 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.013154 10604 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.013691 10262 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.013281 10682 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:17.014501 10682 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.010838 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:17.016487 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 50301.1us
W20240507 08:05:17.028635 10155 leader_election.cc:341] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.045872 10261 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:17.049639 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:16.982803 10231 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.050877 10675 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.059960 10682 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:17.061919 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:17.092900 10191 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:17.062490 10606 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20240507 08:05:17.095520 10382 tablet_bootstrap.cc:654] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:17.090165 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 101524us
W20240507 08:05:17.105000 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 55490.5us
I20240507 08:05:17.073911 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:17.062321 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:17.063963 10188 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:17.065205 10616 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.107283 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:17.118214 10155 leader_election.cc:341] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.123397 10259 raft_consensus.cc:2379] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:17.123245 10687 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240507 08:05:17.125348 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.133081 10686 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.133509 10686 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.133858 10686 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.128294 10190 raft_consensus.cc:2379] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:17.122277 10189 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:17.143785 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:17.149020 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:17.149223 10192 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:17.147498 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.155035 10616 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.158974 10381 tablet_bootstrap.cc:654] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.162797 10232 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:17.164554 10616 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.164945 10616 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.165295 10616 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.183578 10603 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.186210 10685 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:17.186868 10685 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.164657 10232 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:17.367127 10304 consensus_peers.cc:485] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:17.207911 10682 consensus_queue.cc:1035] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.163s
I20240507 08:05:17.371924 10383 tablet_bootstrap.cc:654] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.349808 10688 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.301056 10689 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:17.218417 10676 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.381333 10682 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:17.224403 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 59839.5us
W20240507 08:05:17.406136 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:17.413552 10688 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.425216 10382 tablet_bootstrap.cc:492] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:17.425695 10382 ts_tablet_manager.cc:1367] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.521s user 0.029s sys 0.018s
I20240507 08:05:17.424912 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:17.417847 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:17.432310 10155 leader_election.cc:341] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.426913 10685 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.419370 10616 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.433887 10686 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.435525 10616 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.435736 10686 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:17.425536 10688 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.437711 10190 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:17.438263 10306 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.441875 10696 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:17.442376 10696 raft_consensus.cc:686] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:17.442940 10469 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:17.443845 10686 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:17.415355 10682 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.448001 10381 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:17.448424 10381 ts_tablet_manager.cc:1367] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.449s user 0.033s sys 0.059s
I20240507 08:05:17.443177 10696 consensus_queue.cc:237] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.452009 10682 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:17.417266 10689 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.460224 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:17.460469 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.460992 10259 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:17.461283 10187 raft_consensus.cc:2379] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:17.461282 10382 raft_consensus.cc:348] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.462157 10382 raft_consensus.cc:374] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.462273 10258 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:17.462424 10382 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.463099 10382 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.463577 10676 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:17.464561 10156 leader_election.cc:341] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.464972 10156 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.465236 10382 ts_tablet_manager.cc:1398] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20240507 08:05:17.463104 10689 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.467016 10499 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.457408 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:17.469349 10156 leader_election.cc:341] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.469808 10156 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.470361 10604 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240507 08:05:17.473001 10681 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.474519 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.474678 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.475430 10335 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:17.475492 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.473752 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.477028 10187 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.478963 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:17.479540 10189 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:17.494712 10683 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240507 08:05:17.488852 10407 consensus_queue.cc:1035] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240507 08:05:17.498292 10515 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:17.498800 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:17.502476 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.503298 10185 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.528138 10382 tablet_bootstrap.cc:492] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:17.530645 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:17.543695 10304 leader_election.cc:341] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.545814 10383 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:17.546196 10383 ts_tablet_manager.cc:1367] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.537s user 0.017s sys 0.010s
I20240507 08:05:17.549813 10304 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:17.553053 10231 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.554634 10383 raft_consensus.cc:348] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.555353 10383 raft_consensus.cc:374] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.555625 10383 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.556257 10383 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.557606 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:17.558208 10334 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.544214 10232 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:17.562163 10683 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.562537 10683 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.562826 10683 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:17.563611 10231 leader_election.cc:341] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.566334 10383 ts_tablet_manager.cc:1398] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20240507 08:05:17.566036 10381 raft_consensus.cc:348] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.567453 10381 raft_consensus.cc:374] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.567721 10381 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.568361 10381 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.565557 10606 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.572690 10606 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.574321 10606 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.563779 10689 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.576943 10383 tablet_bootstrap.cc:492] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:17.571188 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.579861 10231 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.581028 10675 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.582175 10187 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.578670 10381 ts_tablet_manager.cc:1398] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.130s user 0.005s sys 0.002s
I20240507 08:05:17.584638 10381 tablet_bootstrap.cc:492] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:17.557193 10688 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.611442 10382 tablet_bootstrap.cc:654] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.622795 10688 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.630934 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:17.635679 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:17.636005 10231 leader_election.cc:341] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:17.626662 10156 leader_election.cc:341] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.624786 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:17.636837 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:17.646350 10156 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.653630 10383 tablet_bootstrap.cc:654] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.626914 10232 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:17.654359 10499 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.655418 10604 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:17.655879 10604 raft_consensus.cc:686] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:17.663548 10689 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:17.665150 10689 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.669814 10339 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:17.680142 10689 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.656644 10604 consensus_queue.cc:237] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:17.694376 10704 log.cc:924] Time spent T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20240507 08:05:17.697613 10090 catalog_manager.cc:5484] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:17.704435 10387 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:17.715438 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.718191 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.726147 10387 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.729722 10191 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.742431 10383 tablet_bootstrap.cc:492] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
W20240507 08:05:17.747510 10703 log.cc:924] Time spent T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240507 08:05:17.731789 10339 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:17.749174 10383 ts_tablet_manager.cc:1367] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.172s user 0.018s sys 0.009s
I20240507 08:05:17.748791 10231 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:17.750686 10689 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.751026 10689 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.751345 10689 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.751247 10306 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:17.752406 10306 leader_election.cc:341] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.754729 10683 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.756675 10683 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:17.757356 10338 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:17.758097 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:17.758723 10387 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.760010 10185 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.753337 10696 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.760627 10689 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.761524 10232 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.762308 10606 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.762835 10689 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.762676 10675 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:17.763502 10258 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:17.737294 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:17.763752 10675 raft_consensus.cc:686] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:17.764014 10190 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:17.764683 10675 consensus_queue.cc:237] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.766746 10191 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.767925 10232 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.770677 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.771086 10381 tablet_bootstrap.cc:654] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.770713 10606 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.771292 10192 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.771677 10606 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:17.784574 10606 raft_consensus.cc:686] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:17.785787 10383 raft_consensus.cc:348] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.786585 10383 raft_consensus.cc:374] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.786854 10383 raft_consensus.cc:729] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.753293 10306 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.788394 10383 consensus_queue.cc:260] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:17.797133 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44781us
I20240507 08:05:17.798518 10383 ts_tablet_manager.cc:1398] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20240507 08:05:17.798699 10469 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.799422 10383 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:17.795082 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:17.796097 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:17.813386 10383 tablet_bootstrap.cc:654] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.811654 10333 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:17.811358 10382 tablet_bootstrap.cc:492] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
W20240507 08:05:17.812279 10155 leader_election.cc:341] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.817260 10382 ts_tablet_manager.cc:1367] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.299s user 0.010s sys 0.011s
I20240507 08:05:17.818423 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:17.818344 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.819173 10338 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.819528 10185 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:17.819629 10382 raft_consensus.cc:348] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.820433 10382 raft_consensus.cc:374] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.820748 10382 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.821342 10382 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.813803 10334 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.807586 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.831864 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:17.838757 10382 ts_tablet_manager.cc:1398] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20240507 08:05:17.839983 10382 tablet_bootstrap.cc:492] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
W20240507 08:05:17.837461 10156 leader_election.cc:341] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.844616 10156 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.828166 10606 consensus_queue.cc:237] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.850965 10192 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.851145 10706 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.852352 10232 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.855304 10675 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:17.855793 10675 raft_consensus.cc:686] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:17.855865 10306 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:17.857546 10683 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:05:17.856827 10675 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.869813 10338 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.871063 10231 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:17.873160 10604 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:17.873818 10604 raft_consensus.cc:686] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:17.874610 10604 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.898623 10382 tablet_bootstrap.cc:654] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.905539 10381 tablet_bootstrap.cc:492] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:17.906055 10381 ts_tablet_manager.cc:1367] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.322s user 0.020s sys 0.001s
I20240507 08:05:17.910625 10381 raft_consensus.cc:348] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.911356 10381 raft_consensus.cc:374] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.911537 10381 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.909422 10089 catalog_manager.cc:5484] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:17.912472 10381 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.917307 10381 ts_tablet_manager.cc:1398] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20240507 08:05:17.918350 10381 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:17.919746 10687 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240507 08:05:17.933548 10381 tablet_bootstrap.cc:654] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.946698 10383 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:17.947681 10383 ts_tablet_manager.cc:1367] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.148s user 0.021s sys 0.028s
I20240507 08:05:17.942230 10185 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:17.951233 10383 raft_consensus.cc:348] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.951921 10383 raft_consensus.cc:374] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.952183 10383 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.952917 10383 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.954957 10687 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:17.955960 10383 ts_tablet_manager.cc:1398] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240507 08:05:17.956933 10383 tablet_bootstrap.cc:492] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:17.964584 10383 tablet_bootstrap.cc:654] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:17.968724 10382 tablet_bootstrap.cc:492] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:17.971740 10706 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:17.972404 10706 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.969405 10382 ts_tablet_manager.cc:1367] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.130s user 0.010s sys 0.009s
I20240507 08:05:17.975909 10706 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:17.976840 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:17.976998 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:17.977782 10267 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
W20240507 08:05:17.979714 10156 leader_election.cc:341] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:17.980500 10155 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:17.980597 10382 raft_consensus.cc:348] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.982498 10382 raft_consensus.cc:374] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:17.983140 10382 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:17.981873 10706 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:17.984301 10706 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:17.984472 10382 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:17.987079 10382 ts_tablet_manager.cc:1398] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20240507 08:05:17.987006 10706 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.003057 10381 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.003664 10381 ts_tablet_manager.cc:1367] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.008s
I20240507 08:05:18.007306 10381 raft_consensus.cc:348] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.008024 10381 raft_consensus.cc:374] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.008291 10381 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.010558 10381 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.024492 10381 ts_tablet_manager.cc:1398] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.021s user 0.003s sys 0.002s
I20240507 08:05:18.023718 10087 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:18.025418 10381 tablet_bootstrap.cc:492] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.025673 10087 catalog_manager.cc:5484] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:18.031057 10381 tablet_bootstrap.cc:654] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.032330 10382 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:18.032791 10087 catalog_manager.cc:5484] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:18.036813 10087 catalog_manager.cc:5484] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:18.034657 10706 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.041039 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:18.041699 10267 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.043056 10381 tablet_bootstrap.cc:492] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.044845 10381 ts_tablet_manager.cc:1367] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20240507 08:05:18.050025 10267 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:18.053058 10155 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:18.053998 10706 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:18.059695 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:18.054342 10621 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:18.061429 10621 raft_consensus.cc:686] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:18.062279 10621 consensus_queue.cc:237] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:18.063444 10156 leader_election.cc:341] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.067696 10382 tablet_bootstrap.cc:654] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.068259 10383 tablet_bootstrap.cc:492] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:18.059237 10381 raft_consensus.cc:348] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.068742 10383 ts_tablet_manager.cc:1367] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.112s user 0.020s sys 0.037s
I20240507 08:05:18.068886 10381 raft_consensus.cc:374] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.069204 10381 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.069931 10381 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.072053 10381 ts_tablet_manager.cc:1398] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20240507 08:05:18.072876 10381 tablet_bootstrap.cc:492] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.074950 10383 raft_consensus.cc:348] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.075911 10383 raft_consensus.cc:374] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.076239 10383 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.080382 10381 tablet_bootstrap.cc:654] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.077050 10383 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.088299 10383 ts_tablet_manager.cc:1398] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.019s user 0.005s sys 0.002s
I20240507 08:05:18.089473 10383 tablet_bootstrap.cc:492] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:18.096088 10382 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:18.096678 10382 ts_tablet_manager.cc:1367] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.025s
I20240507 08:05:18.096858 10381 tablet_bootstrap.cc:492] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.097333 10381 ts_tablet_manager.cc:1367] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.003s
I20240507 08:05:18.097950 10383 tablet_bootstrap.cc:654] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.099440 10382 raft_consensus.cc:348] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.100339 10382 raft_consensus.cc:374] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.100697 10382 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.099671 10381 raft_consensus.cc:348] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.101327 10381 raft_consensus.cc:374] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.101647 10381 raft_consensus.cc:729] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.102247 10381 consensus_queue.cc:260] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.104241 10381 ts_tablet_manager.cc:1398] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240507 08:05:18.105029 10381 tablet_bootstrap.cc:492] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.111961 10381 tablet_bootstrap.cc:654] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.116537 10382 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.119972 10382 ts_tablet_manager.cc:1398] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20240507 08:05:18.121866 10382 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:18.125334 10086 catalog_manager.cc:5484] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:18.130211 10621 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.132643 10621 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.134580 10621 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:18.137276 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:18.139920 10338 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:18.137259 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.141603 10266 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:18.142940 10156 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:18.143613 10621 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:18.143980 10621 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:18.144294 10621 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.144477 10381 tablet_bootstrap.cc:492] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.146044 10381 ts_tablet_manager.cc:1367] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.005s
I20240507 08:05:18.145596 10382 tablet_bootstrap.cc:654] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.140985 10469 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.149513 10469 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.153963 10621 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.154160 10381 raft_consensus.cc:348] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.154870 10381 raft_consensus.cc:374] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.155191 10381 raft_consensus.cc:729] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.155841 10381 consensus_queue.cc:260] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.158035 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:18.158564 10621 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:18.158690 10259 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.160413 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:18.160986 10338 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.161842 10683 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.162269 10683 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.165057 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.166208 10683 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:18.166934 10304 leader_election.cc:341] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.168378 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.169952 10381 ts_tablet_manager.cc:1398] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.024s user 0.005s sys 0.001s
I20240507 08:05:18.171655 10259 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:18.172055 10338 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:18.173122 10155 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:18.173882 10304 leader_election.cc:341] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.174616 10621 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:18.175112 10621 raft_consensus.cc:686] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:18.175608 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.175714 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.176265 10185 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:18.176085 10621 consensus_queue.cc:237] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.177191 10192 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:18.177772 10306 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.178903 10381 tablet_bootstrap.cc:492] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.179411 10687 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:18.179829 10687 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:18.157099 10469 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.182880 10306 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.184145 10469 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:18.187381 10381 tablet_bootstrap.cc:654] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.180178 10687 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.184542 10469 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:18.190199 10469 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.200059 10383 tablet_bootstrap.cc:492] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:18.200624 10383 ts_tablet_manager.cc:1367] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.111s user 0.019s sys 0.019s
I20240507 08:05:18.204645 10383 raft_consensus.cc:348] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.205394 10383 raft_consensus.cc:374] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.206144 10383 raft_consensus.cc:729] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.206184 10469 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.207202 10383 consensus_queue.cc:260] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.209419 10383 ts_tablet_manager.cc:1398] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240507 08:05:18.210508 10383 tablet_bootstrap.cc:492] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:18.213887 10689 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.214363 10689 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.216464 10689 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:18.218251 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.221761 10687 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.222489 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:18.222569 10383 tablet_bootstrap.cc:654] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.260627 10382 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:18.262228 10382 ts_tablet_manager.cc:1367] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.030s
I20240507 08:05:18.262769 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:18.263797 10259 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.230001 10266 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.269780 10231 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:18.270843 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:18.225610 10706 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.271929 10192 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.271848 10706 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.273069 10259 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:18.265347 10382 raft_consensus.cc:348] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.246501 10090 catalog_manager.cc:5484] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:18.286604 10382 raft_consensus.cc:374] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.286901 10382 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.275580 10706 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:18.275828 10304 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:18.276165 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.278232 10381 tablet_bootstrap.cc:492] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.278914 10192 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:18.294713 10381 ts_tablet_manager.cc:1367] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.116s user 0.030s sys 0.017s
I20240507 08:05:18.225122 10684 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.295281 10515 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:18.295239 10684 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.295869 10515 raft_consensus.cc:686] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:18.228533 10689 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20240507 08:05:18.296660 10515 consensus_queue.cc:237] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.297235 10684 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.297210 10381 raft_consensus.cc:348] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.298429 10381 raft_consensus.cc:374] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.280844 10469 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.298733 10381 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.300024 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.299517 10381 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:18.301121 10155 leader_election.cc:341] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.301908 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:18.302573 10381 ts_tablet_manager.cc:1398] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20240507 08:05:18.303454 10381 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.305259 10469 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:18.305954 10304 leader_election.cc:341] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.307988 10689 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.308764 10381 tablet_bootstrap.cc:654] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:18.308995 10304 leader_election.cc:341] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.309188 10689 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.309881 10683 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.310299 10683 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.312119 10683 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.313707 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.314527 10259 raft_consensus.cc:2379] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 1: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:18.317759 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.305891 10469 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.318408 10186 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:18.319808 10469 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.329092 10689 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.338110 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.342972 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.343389 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.345723 10381 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.346233 10381 ts_tablet_manager.cc:1367] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.002s
I20240507 08:05:18.283959 10383 tablet_bootstrap.cc:492] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:18.285748 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:18.346827 10383 ts_tablet_manager.cc:1367] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.007s
W20240507 08:05:18.348057 10156 leader_election.cc:341] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.348484 10156 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.349211 10383 raft_consensus.cc:348] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.349927 10383 raft_consensus.cc:374] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.350189 10383 raft_consensus.cc:729] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.350759 10383 consensus_queue.cc:260] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.285871 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:18.352188 10185 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.287843 10382 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.281859 10687 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.346071 10306 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.354061 10382 ts_tablet_manager.cc:1398] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.092s user 0.009s sys 0.004s
I20240507 08:05:18.354938 10382 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:18.347957 10090 catalog_manager.cc:5484] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:18.354046 10706 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:18.362785 10687 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:18.364189 10687 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:18.364336 10606 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:18.364537 10687 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.365720 10306 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.367820 10696 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.366919 10306 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.377944 10469 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:18.389039 10339 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.423058 10689 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:18.462742 10621 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.463258 10621 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.490841 10381 raft_consensus.cc:348] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.491446 10689 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.500926 10381 raft_consensus.cc:374] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.501341 10381 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.502048 10381 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.494992 10382 tablet_bootstrap.cc:654] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.495215 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.495543 10383 ts_tablet_manager.cc:1398] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.148s user 0.016s sys 0.047s
I20240507 08:05:18.508208 10383 tablet_bootstrap.cc:492] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:18.508723 10381 ts_tablet_manager.cc:1398] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.157s user 0.010s sys 0.015s
I20240507 08:05:18.509707 10381 tablet_bootstrap.cc:492] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.511754 10689 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.522938 10383 tablet_bootstrap.cc:654] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.523370 10689 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:05:18.526504 10748 log.cc:924] Time spent T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
W20240507 08:05:18.530234 10155 leader_election.cc:341] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.530568 10696 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:18.536783 10621 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:18.537786 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:18.539103 10156 leader_election.cc:341] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.539510 10156 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.544333 10706 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:05:18.541532 10745 log.cc:924] Time spent T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.082s user 0.000s sys 0.005s
I20240507 08:05:18.544922 10185 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:18.567055 10382 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:18.567598 10382 ts_tablet_manager.cc:1367] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.213s user 0.028s sys 0.057s
W20240507 08:05:18.575057 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 208295us
I20240507 08:05:18.576105 10306 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:18.579612 10383 tablet_bootstrap.cc:492] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:18.580070 10383 ts_tablet_manager.cc:1367] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.008s
I20240507 08:05:18.577993 10382 raft_consensus.cc:348] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.581009 10382 raft_consensus.cc:374] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.581285 10382 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.582033 10382 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.583176 10381 tablet_bootstrap.cc:654] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.586155 10683 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:18.594030 10683 raft_consensus.cc:686] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:18.611941 10383 raft_consensus.cc:348] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.612706 10383 raft_consensus.cc:374] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.612983 10383 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
W20240507 08:05:18.616058 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:18.613569 10383 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.624825 10382 ts_tablet_manager.cc:1398] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.057s user 0.006s sys 0.002s
I20240507 08:05:18.625543 10382 tablet_bootstrap.cc:492] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:18.637435 10383 ts_tablet_manager.cc:1398] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.057s user 0.007s sys 0.000s
I20240507 08:05:18.640276 10383 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:18.632442 10696 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.648947 10696 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.631912 10620 consensus_queue.cc:1035] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240507 08:05:18.660964 10382 tablet_bootstrap.cc:654] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.665419 10383 tablet_bootstrap.cc:654] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:18.655035 10687 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.652086 10683 consensus_queue.cc:237] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.683622 10758 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
W20240507 08:05:18.713258 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 59624.9us
I20240507 08:05:18.718833 10687 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.720546 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:18.721959 10304 leader_election.cc:341] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.725479 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.726181 10266 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:18.727203 10304 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:18.732162 10687 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:18.732564 10687 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:18.732941 10687 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.735304 10696 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.736047 10767 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.732374 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:18.744485 10621 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240507 08:05:18.751699 10621 consensus_queue.cc:1035] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:05:18.788273 10306 leader_election.cc:341] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:18.811542 10762 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:18.811543 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:18.814224 10186 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:18.824338 10381 tablet_bootstrap.cc:492] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:18.825055 10381 ts_tablet_manager.cc:1367] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.316s user 0.011s sys 0.015s
I20240507 08:05:18.762168 10687 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.862166 10381 raft_consensus.cc:348] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.863220 10381 raft_consensus.cc:374] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:18.863569 10381 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:18.864427 10381 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:18.828788 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 43552.9us
I20240507 08:05:18.888806 10381 ts_tablet_manager.cc:1398] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.063s user 0.006s sys 0.001s
I20240507 08:05:18.898345 10381 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:18.910758 10515 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.918960 10759 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.961293 10382 tablet_bootstrap.cc:492] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:18.962652 10382 ts_tablet_manager.cc:1367] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.337s user 0.038s sys 0.024s
I20240507 08:05:18.926128 10761 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
W20240507 08:05:18.965960 10770 log.cc:924] Time spent T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20240507 08:05:18.968773 10687 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.960314 10759 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.959151 10762 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:18.959218 10515 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.974088 10515 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:18.980803 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:18.987361 10383 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:18.987916 10383 ts_tablet_manager.cc:1367] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.348s user 0.024s sys 0.013s
I20240507 08:05:18.988778 10762 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:18.994547 10186 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:19.014751 10381 tablet_bootstrap.cc:654] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.054245 10383 raft_consensus.cc:348] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.055313 10383 raft_consensus.cc:374] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.055785 10383 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:19.054968 10759 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.057029 10383 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.059687 10382 raft_consensus.cc:348] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.060246 10383 ts_tablet_manager.cc:1398] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.072s user 0.010s sys 0.002s
I20240507 08:05:19.060467 10382 raft_consensus.cc:374] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.060792 10382 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:19.061434 10382 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:19.076304 10304 leader_election.cc:341] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.064359 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:19.083109 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:19.097940 10382 ts_tablet_manager.cc:1398] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.135s user 0.006s sys 0.000s
I20240507 08:05:19.098690 10382 tablet_bootstrap.cc:492] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:19.086810 10258 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.216318 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:19.221477 10383 tablet_bootstrap.cc:492] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:19.103405 10760 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.387526 10760 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.388605 10383 tablet_bootstrap.cc:654] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.123916 10381 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:19.399902 10381 ts_tablet_manager.cc:1367] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.510s user 0.015s sys 0.010s
I20240507 08:05:19.097349 10762 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.126987 10382 tablet_bootstrap.cc:654] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.300813 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:19.406358 10337 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
I20240507 08:05:19.307791 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:19.407919 10191 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:19.408387 10758 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.328843 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:19.408892 10758 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.368530 10258 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:19.417760 10758 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.418049 10381 raft_consensus.cc:348] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.393590 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:19.396274 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 84756.2us
W20240507 08:05:19.422377 10231 leader_election.cc:341] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.357730 10306 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.412249 10156 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:19.426127 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 69338.5us
I20240507 08:05:19.426446 10763 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:19.426990 10763 raft_consensus.cc:686] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:19.428333 10762 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:19.427875 10763 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.428709 10762 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:19.429085 10762 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.432641 10706 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.433096 10706 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.434999 10706 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.421046 10760 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:19.369208 10087 catalog_manager.cc:5484] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:19.436527 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:19.436750 10259 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 0.
W20240507 08:05:19.437883 10156 leader_election.cc:341] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.438694 10762 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.443744 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:19.444322 10334 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.449632 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:19.450798 10334 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:19.451010 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:19.451674 10338 raft_consensus.cc:2379] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 1: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:19.452329 10156 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:19.453119 10706 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:19.453784 10706 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:19.454598 10706 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.459029 10381 raft_consensus.cc:374] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.459470 10381 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:19.460307 10381 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.465471 10762 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.497473 10381 ts_tablet_manager.cc:1398] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.097s user 0.008s sys 0.000s
I20240507 08:05:19.498837 10381 tablet_bootstrap.cc:492] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
W20240507 08:05:19.527609 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 99030us
W20240507 08:05:19.532910 10306 leader_election.cc:341] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.533732 10306 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.507551 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:19.536115 10687 raft_consensus.cc:2741] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240507 08:05:19.548009 10304 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:19.538600 10763 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20240507 08:05:19.555932 10760 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:19.556638 10760 raft_consensus.cc:686] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:19.557797 10760 consensus_queue.cc:237] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.566355 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:19.567049 10258 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:19.571424 10304 consensus_peers.cc:485] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:19.574646 10381 tablet_bootstrap.cc:654] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.580749 10382 tablet_bootstrap.cc:492] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:19.581240 10382 ts_tablet_manager.cc:1367] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.483s user 0.008s sys 0.028s
I20240507 08:05:19.582455 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:19.583091 10186 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:19.583675 10382 raft_consensus.cc:348] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.584365 10382 raft_consensus.cc:374] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.584606 10382 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
W20240507 08:05:19.535648 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 133515us
W20240507 08:05:19.591840 10232 leader_election.cc:341] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.592358 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:19.592752 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:19.593631 10382 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:19.601764 10304 leader_election.cc:341] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.600064 10797 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:19.609998 10382 ts_tablet_manager.cc:1398] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.028s user 0.008s sys 0.000s
W20240507 08:05:19.610186 10155 leader_election.cc:341] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.610661 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.593216 10683 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240507 08:05:19.609526 10797 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:19.615648 10797 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.615715 10794 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:19.614382 10685 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:19.668444 10383 tablet_bootstrap.cc:492] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:19.673357 10383 ts_tablet_manager.cc:1367] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.452s user 0.069s sys 0.066s
I20240507 08:05:19.685170 10258 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:19.686916 10382 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:19.703856 10382 tablet_bootstrap.cc:654] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.714936 10383 raft_consensus.cc:348] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.723594 10383 raft_consensus.cc:374] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.723992 10383 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:19.724705 10383 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:19.739143 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:19.749516 10383 ts_tablet_manager.cc:1398] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20240507 08:05:19.752010 10792 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.752691 10792 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.757105 10763 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.757632 10763 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.759536 10763 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:19.763192 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:19.764228 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:19.764870 10183 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
W20240507 08:05:19.766454 10304 leader_election.cc:341] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.767694 10797 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.771685 10306 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.772378 10383 tablet_bootstrap.cc:492] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:19.775611 10683 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:19.781556 10797 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:19.787257 10683 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:19.793550 10683 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.795039 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:19.797901 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:19.798256 10259 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.801389 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:19.805033 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:19.810158 10232 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:19.810968 10759 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:05:19.811647 10155 leader_election.cc:341] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.784016 10792 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.814535 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:19.815645 10183 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:05:19.815125 10156 leader_election.cc:341] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.816545 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.820788 10762 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:19.826865 10706 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.836897 10706 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.839010 10706 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.842994 10381 tablet_bootstrap.cc:492] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:19.843489 10381 ts_tablet_manager.cc:1367] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.345s user 0.039s sys 0.021s
I20240507 08:05:19.845729 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:19.849310 10156 leader_election.cc:341] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.861682 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:19.865370 10155 leader_election.cc:341] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.865821 10155 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.866161 10763 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:19.853463 10381 raft_consensus.cc:348] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.871822 10381 raft_consensus.cc:374] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.872057 10381 raft_consensus.cc:729] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:19.871017 10683 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.872752 10381 consensus_queue.cc:260] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.874311 10183 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:19.873194 10791 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:19.875425 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:19.876906 10683 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:19.877772 10382 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:19.879504 10383 tablet_bootstrap.cc:654] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.880353 10382 ts_tablet_manager.cc:1367] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.194s user 0.010s sys 0.034s
I20240507 08:05:19.879822 10381 ts_tablet_manager.cc:1398] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20240507 08:05:19.884744 10797 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:19.885545 10382 raft_consensus.cc:348] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.886337 10382 raft_consensus.cc:374] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:19.886651 10382 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:19.888581 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:19.889773 10381 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:19.886895 10259 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:19.891786 10382 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:19.891988 10304 leader_election.cc:341] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.894033 10797 raft_consensus.cc:686] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:19.896236 10797 consensus_queue.cc:237] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.889622 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:19.899008 10185 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:19.902868 10621 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.910041 10621 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.913201 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:19.916685 10155 leader_election.cc:341] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.917207 10792 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:19.921861 10621 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:19.931890 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:19.933373 10760 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:19.933931 10760 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:19.935752 10381 tablet_bootstrap.cc:654] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
W20240507 08:05:19.933396 10156 leader_election.cc:341] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:19.936379 10156 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:19.936749 10382 ts_tablet_manager.cc:1398] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.056s user 0.005s sys 0.017s
I20240507 08:05:19.937795 10382 tablet_bootstrap.cc:492] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:19.947659 10382 tablet_bootstrap.cc:654] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:19.951872 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:19.939214 10792 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:19.952544 10259 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:19.941130 10760 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:20.073853 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:20.073143 10304 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:20.098624 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 141422us
I20240507 08:05:20.100327 10780 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:20.100772 10780 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:20.101161 10780 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.103973 10185 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:20.106494 10382 tablet_bootstrap.cc:492] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:20.106966 10382 ts_tablet_manager.cc:1367] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.169s user 0.020s sys 0.010s
I20240507 08:05:20.114708 10621 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:20.109421 10382 raft_consensus.cc:348] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.118964 10382 raft_consensus.cc:374] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.119243 10382 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.119947 10382 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.121502 10382 ts_tablet_manager.cc:1398] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20240507 08:05:20.122336 10382 tablet_bootstrap.cc:492] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: Bootstrap starting.
I20240507 08:05:20.124855 10383 tablet_bootstrap.cc:492] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:20.125262 10383 ts_tablet_manager.cc:1367] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.374s user 0.008s sys 0.004s
I20240507 08:05:20.129316 10383 raft_consensus.cc:348] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.130035 10383 raft_consensus.cc:374] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.130301 10383 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.130946 10383 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.135325 10382 tablet_bootstrap.cc:654] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:20.135679 10383 ts_tablet_manager.cc:1398] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240507 08:05:20.136866 10381 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:20.137144 10383 tablet_bootstrap.cc:492] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
W20240507 08:05:20.147876 10306 leader_election.cc:341] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:05:20.149641 10304 consensus_peers.cc:485] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:20.167308 10306 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:20.164021 10086 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:20.167516 10381 ts_tablet_manager.cc:1367] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.278s user 0.016s sys 0.003s
I20240507 08:05:20.141520 10780 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.171262 10760 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:20.171768 10760 raft_consensus.cc:686] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:20.171977 10780 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:20.170207 10086 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:05:20.172927 10760 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.150044 10382 tablet_bootstrap.cc:492] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: No bootstrap required, opened a new log
I20240507 08:05:20.178175 10382 ts_tablet_manager.cc:1367] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.003s
I20240507 08:05:20.180497 10382 raft_consensus.cc:348] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.181169 10382 raft_consensus.cc:374] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.183233 10185 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.184899 10383 tablet_bootstrap.cc:654] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:20.181440 10382 raft_consensus.cc:729] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.194561 10382 consensus_queue.cc:260] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.196206 10382 ts_tablet_manager.cc:1398] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240507 08:05:20.227221 10791 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:20.227892 10791 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:20.274253 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08000a2ce0 after lost signal to thread 9049
I20240507 08:05:20.373005 10381 raft_consensus.cc:348] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.377715 10381 raft_consensus.cc:374] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.378005 10381 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.393349 10381 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:20.412680 10362 debug-util.cc:398] Leaking SignalData structure 0x7b0800058fc0 after lost signal to thread 10124
I20240507 08:05:20.427448 10381 ts_tablet_manager.cc:1398] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.259s user 0.005s sys 0.003s
I20240507 08:05:20.439196 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:20.439908 10191 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.446640 10791 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.470777 10381 tablet_bootstrap.cc:492] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
W20240507 08:05:20.454923 10362 debug-util.cc:398] Leaking SignalData structure 0x7b080004a340 after lost signal to thread 10216
W20240507 08:05:20.505890 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 95881.9us
I20240507 08:05:20.508750 10767 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:20.517290 10362 debug-util.cc:398] Leaking SignalData structure 0x7b0800238240 after lost signal to thread 10291
I20240507 08:05:20.510192 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:20.545709 10383 tablet_bootstrap.cc:492] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:20.546211 10383 ts_tablet_manager.cc:1367] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.409s user 0.036s sys 0.052s
I20240507 08:05:20.550892 10258 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.524267 10687 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20240507 08:05:20.541404 10767 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:20.565625 10362 debug-util.cc:398] Leaking SignalData structure 0x7b0800281360 after lost signal to thread 10365
I20240507 08:05:20.587530 10381 tablet_bootstrap.cc:654] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:20.580749 10829 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:20.591039 10829 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.591833 10684 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:20.592429 10684 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.593050 10829 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.593890 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:20.595064 10684 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:20.595964 10383 raft_consensus.cc:348] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.596649 10383 raft_consensus.cc:374] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.596927 10383 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.601217 10191 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:20.602596 10383 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.632318 10306 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:20.639160 10258 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:20.639448 10763 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:20.645777 10763 raft_consensus.cc:686] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:20.646625 10763 consensus_queue.cc:237] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.648784 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:20.649981 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:20.640502 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:20.628093 10827 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20240507 08:05:20.631711 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:20.651607 10383 ts_tablet_manager.cc:1398] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.090s user 0.007s sys 0.000s
W20240507 08:05:20.686009 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 55912us
I20240507 08:05:20.660043 10192 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
W20240507 08:05:20.692116 10155 leader_election.cc:341] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:20.696774 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:20.697769 10189 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:20.683787 10832 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:20.678252 10407 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240507 08:05:20.700796 10832 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.702814 10832 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.704772 10232 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:20.718461 10306 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:20.721904 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:20.722590 10264 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:20.727164 10155 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:20.728713 10835 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:20.729091 10835 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:20.710142 10794 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:20.731029 10383 tablet_bootstrap.cc:492] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:20.731256 10794 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:20.731662 10794 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.685549 10687 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20240507 08:05:20.729408 10835 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:20.738106 10684 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:20.738519 10684 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:20.738862 10684 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.740224 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:20.747637 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 60143.5us
W20240507 08:05:20.751430 10304 leader_election.cc:341] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:20.755616 10381 tablet_bootstrap.cc:492] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:20.756119 10381 ts_tablet_manager.cc:1367] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.301s user 0.012s sys 0.009s
I20240507 08:05:20.758555 10381 raft_consensus.cc:348] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.759249 10381 raft_consensus.cc:374] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.759513 10381 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.760226 10381 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.764515 10381 ts_tablet_manager.cc:1398] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20240507 08:05:20.765424 10381 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:20.768620 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:20.769286 10335 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:20.770323 10156 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:20.772615 10829 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:20.784674 10383 tablet_bootstrap.cc:654] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:20.786314 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:20.790202 10767 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.798187 10684 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.799723 10339 raft_consensus.cc:2379] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 1: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
W20240507 08:05:20.813258 10304 consensus_peers.cc:485] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Couldn't send request to peer bcf08c8276ba4a9ba646ad0f7090fcd8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:20.816578 10763 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:20.817914 10794 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.829413 10381 tablet_bootstrap.cc:654] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:20.856978 10156 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:20.863307 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:20.864066 10835 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.866297 10835 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.867381 10190 raft_consensus.cc:1226] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20240507 08:05:20.872287 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:20.859642 10087 catalog_manager.cc:5484] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:20.874900 10087 catalog_manager.cc:5484] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:20.877367 10087 catalog_manager.cc:5484] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
W20240507 08:05:20.878739 10138 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20240507 08:05:20.871731 10846 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.879127 10381 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:20.879380 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:20.885529 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:20.873011 10684 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:20.886821 10304 leader_election.cc:341] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:20.888825 10381 ts_tablet_manager.cc:1367] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.124s user 0.010s sys 0.004s
I20240507 08:05:20.892072 10383 tablet_bootstrap.cc:492] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:20.892590 10383 ts_tablet_manager.cc:1367] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.162s user 0.028s sys 0.018s
I20240507 08:05:20.893651 10684 consensus_queue.cc:1046] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:20.895031 10156 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:20.895242 10383 raft_consensus.cc:348] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.896171 10383 raft_consensus.cc:374] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.896333 10829 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:20.896505 10383 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.900257 10830 raft_consensus.cc:3032] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:20.901741 10830 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:20.902577 10830 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.903962 10830 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:20.897459 10383 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.907289 10383 ts_tablet_manager.cc:1398] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240507 08:05:20.908211 10383 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:20.913084 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:20.913751 10183 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.917937 10794 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.919400 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:20.920022 10339 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.930724 10339 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:20.933382 10231 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:20.934583 10794 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:20.935217 10794 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:20.936017 10794 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.946393 10767 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:20.946895 10767 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.951115 10381 raft_consensus.cc:348] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.952284 10381 raft_consensus.cc:374] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:20.952639 10381 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:20.954355 10381 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.962561 10767 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:20.963672 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:20.964365 10188 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:20.966079 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:20.967319 10767 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:20.967789 10767 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:20.968199 10767 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:20.969494 10383 tablet_bootstrap.cc:654] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:20.972612 10183 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:20.984160 10381 ts_tablet_manager.cc:1398] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.095s user 0.005s sys 0.003s
I20240507 08:05:20.982899 10842 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:20.986297 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:20.988731 10231 leader_election.cc:341] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:20.987900 10842 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:20.992619 10846 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:20.993372 10767 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.001854 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:21.002504 10183 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.007052 10381 tablet_bootstrap.cc:492] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:21.009518 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:21.011543 10767 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:21.005281 10186 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
W20240507 08:05:21.012874 10231 leader_election.cc:341] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:21.014353 10261 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.017009 10306 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:21.017751 10780 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:21.018215 10780 raft_consensus.cc:686] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:21.018975 10780 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.022387 10842 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:21.034675 10381 tablet_bootstrap.cc:654] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:21.036470 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:21.038393 10183 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:21.040222 10383 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
W20240507 08:05:21.040407 10304 leader_election.cc:341] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:21.040855 10383 ts_tablet_manager.cc:1367] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.133s user 0.017s sys 0.001s
I20240507 08:05:21.042240 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:21.042896 10186 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 0.
I20240507 08:05:21.045913 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:21.050021 10794 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:21.050572 10794 raft_consensus.cc:686] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:21.051354 10794 consensus_queue.cc:237] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.051800 10383 raft_consensus.cc:348] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.052573 10383 raft_consensus.cc:374] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.053102 10383 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.055191 10383 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.062144 10383 ts_tablet_manager.cc:1398] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20240507 08:05:21.064811 10383 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:21.046242 10306 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:21.068368 10760 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:21.068737 10760 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:21.069056 10760 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.079090 10087 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:21.129698 10090 catalog_manager.cc:5484] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:21.131536 10760 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.144604 10830 consensus_queue.cc:1035] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.145448 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:21.145978 10760 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:21.146096 10188 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.147689 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:21.151058 10304 leader_election.cc:341] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:05:21.151152 10383 tablet_bootstrap.cc:654] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:21.139628 10381 tablet_bootstrap.cc:492] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:21.154090 10381 ts_tablet_manager.cc:1367] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.147s user 0.013s sys 0.021s
I20240507 08:05:21.156106 10188 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:21.157189 10306 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:21.158424 10830 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:21.158926 10830 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:21.159802 10830 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.157189 10381 raft_consensus.cc:348] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.167423 10381 raft_consensus.cc:374] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.167688 10381 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.175843 10381 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.178784 10381 ts_tablet_manager.cc:1398] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.024s user 0.004s sys 0.001s
I20240507 08:05:21.179656 10381 tablet_bootstrap.cc:492] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: Bootstrap starting.
I20240507 08:05:21.187528 10383 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:21.189359 10381 tablet_bootstrap.cc:654] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:21.189852 10760 consensus_queue.cc:1035] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.199064 10383 ts_tablet_manager.cc:1367] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.134s user 0.003s sys 0.009s
I20240507 08:05:21.198467 10087 catalog_manager.cc:5484] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:21.209317 10383 raft_consensus.cc:348] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.210105 10383 raft_consensus.cc:374] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.210373 10383 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.210968 10383 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.219125 10383 ts_tablet_manager.cc:1398] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20240507 08:05:21.220012 10383 tablet_bootstrap.cc:492] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:21.220783 10381 tablet_bootstrap.cc:492] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: No bootstrap required, opened a new log
I20240507 08:05:21.216048 10090 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:21.221223 10381 ts_tablet_manager.cc:1367] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.009s
I20240507 08:05:21.224721 10381 raft_consensus.cc:348] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.228037 10381 raft_consensus.cc:374] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.228288 10381 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.228842 10381 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.222019 10090 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:21.231094 10381 ts_tablet_manager.cc:1398] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20240507 08:05:21.237064 10383 tablet_bootstrap.cc:654] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:21.244397 10383 tablet_bootstrap.cc:492] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:21.244839 10383 ts_tablet_manager.cc:1367] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20240507 08:05:21.247184 10383 raft_consensus.cc:348] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.247884 10383 raft_consensus.cc:374] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.248143 10383 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.248726 10383 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.258708 10383 ts_tablet_manager.cc:1398] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20240507 08:05:21.259562 10383 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:21.269896 10383 tablet_bootstrap.cc:654] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:21.275779 10383 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:21.276268 10383 ts_tablet_manager.cc:1367] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240507 08:05:21.279174 10383 raft_consensus.cc:348] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.281306 10383 raft_consensus.cc:374] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.281618 10383 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.282213 10383 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.284232 10383 ts_tablet_manager.cc:1398] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240507 08:05:21.285697 10383 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Bootstrap starting.
I20240507 08:05:21.290702 10383 tablet_bootstrap.cc:654] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Neither blocks nor log segments found. Creating new log.
I20240507 08:05:21.299618 10383 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: No bootstrap required, opened a new log
I20240507 08:05:21.300005 10383 ts_tablet_manager.cc:1367] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.003s
I20240507 08:05:21.301919 10383 raft_consensus.cc:348] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.302606 10383 raft_consensus.cc:374] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:05:21.302870 10383 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Initialized, Role: FOLLOWER
I20240507 08:05:21.303512 10383 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.305624 10383 ts_tablet_manager.cc:1398] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240507 08:05:21.354099 10833 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.356709 10833 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.365428 10259 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.415113 10758 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:21.415606 10758 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.417523 10758 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:21.419113 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:21.419986 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:21.421636 10337 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:21.422787 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:21.423583 10833 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:05:21.423923 10833 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:21.424232 10833 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.432541 10833 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.433601 10188 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 0.
I20240507 08:05:21.436498 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:21.436863 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:21.437088 10188 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.437553 10337 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.441192 10261 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.440048 10833 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:21.447068 10188 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:21.449605 10232 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:21.450459 10833 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:05:21.450965 10833 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:21.451799 10833 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.455116 10842 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:21.458060 10337 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:21.472957 10760 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.489610 10767 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:21.490159 10767 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:21.492913 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:21.495128 10767 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:21.495359 10263 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.499584 10758 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.519671 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:21.531657 10339 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.547036 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:21.574673 10767 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.623165 10759 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:05:21.634616 10087 catalog_manager.cc:5484] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:21.645768 10781 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.658795 10263 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:05:21.672214 10781 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:21.839179 10759 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:21.896481 10759 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:05:22.344606 10881 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:22.345392 10881 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:22.347896 10881 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:22.348852 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:22.349083 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:22.349884 10188 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:22.352202 10306 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:22.352972 10881 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:22.353338 10881 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:22.353624 10881 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:22.359167 10881 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:22.360801 10881 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:22.362221 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:22.361408 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:22.363086 10188 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:22.370311 10188 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:22.371052 10306 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:22.371754 10880 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:22.372265 10880 raft_consensus.cc:686] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:22.373252 10880 consensus_queue.cc:237] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:22.383390 10767 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:22.383949 10767 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:22.386049 10767 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:22.386878 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:22.387704 10188 raft_consensus.cc:2354] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:22.387039 10090 catalog_manager.cc:5484] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:22.387718 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:22.389202 10232 leader_election.cc:398] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:22.389541 10232 leader_election.cc:401] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:05:22.390583 10767 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:22.402804 10767 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:22.862689 10259 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:22.863958 10881 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:22.880159 10186 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:22.881531 10842 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:05:23.605608 9046 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:05:23.637006 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:23.641949 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:05:23.644645 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:05:23.647447 9046 server_base.cc:1034] running on GCE node
I20240507 08:05:23.648778 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240507 08:05:23.649097 9046 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240507 08:05:23.649323 9046 hybrid_clock.cc:648] HybridClock initialized: now 1715069123649307 us; error 0 us; skew 500 ppm
I20240507 08:05:23.650228 9046 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:05:23.654186 9046 webserver.cc:460] Webserver started at http://127.8.213.132:35415/ using document root <none> and password file <none>
I20240507 08:05:23.654976 9046 fs_manager.cc:359] Metadata directory not provided
I20240507 08:05:23.655346 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:05:23.655850 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:05:23.658026 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-3-root/instance:
uuid: "5d6656a616b747d393ff3de5328d2540"
format_stamp: "Formatted at 2024-05-07 08:05:23 on dist-test-slave-ls56"
I20240507 08:05:23.664882 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20240507 08:05:23.669817 10905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:23.670720 9046 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240507 08:05:23.671046 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-3-root
uuid: "5d6656a616b747d393ff3de5328d2540"
format_stamp: "Formatted at 2024-05-07 08:05:23 on dist-test-slave-ls56"
I20240507 08:05:23.671329 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:05:23.692915 9046 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:05:23.694358 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:05:23.696161 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:05:23.699297 9046 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:05:23.699574 9046 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:23.699939 9046 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:05:23.700121 9046 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:05:23.835779 9046 rpc_server.cc:307] RPC server started. Bound to: 127.8.213.132:44849
I20240507 08:05:23.836037 10967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.213.132:44849 every 8 connection(s)
I20240507 08:05:23.847169 9046 auto_leader_rebalancer-test.cc:405] add a tserver: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:23.859920 10968 heartbeater.cc:347] Connected to a master server at 127.8.213.190:37115
I20240507 08:05:23.860344 10968 heartbeater.cc:464] Registering TS with master...
I20240507 08:05:23.861079 10968 heartbeater.cc:511] Master 127.8.213.190:37115 requested a full tablet report, sending...
I20240507 08:05:23.862836 10090 ts_manager.cc:194] Registered new tserver with Master: 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:23.864194 10090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47176
I20240507 08:05:24.166879 10258 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:24.176649 10186 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.177995 10895 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:24.179744 10339 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.181494 10794 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:24.192009 10895 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.196487 10339 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.214814 10183 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
W20240507 08:05:24.273258 10233 consensus_peers.cc:485] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d0e1a7b6db34cea8e152b755b52aa96. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:24.276154 10090 catalog_manager.cc:5484] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
W20240507 08:05:24.291481 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 51642.4us
I20240507 08:05:24.300011 10259 consensus_queue.cc:237] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
W20240507 08:05:24.314860 10233 consensus_peers.cc:485] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 50d914d033cf48c389262dd8b7c7fbce. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:24.318286 10188 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.320360 10339 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.320533 10982 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:24.321780 10895 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:24.398772 10982 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.426568 10188 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.426658 10337 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.454425 10086 catalog_manager.cc:5484] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:24.475000 10337 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:24.483552 10258 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.484915 10880 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:24.485167 10188 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.487705 10972 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240507 08:05:24.501358 10307 consensus_peers.cc:485] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcbe630c1736458898143fb230a9ffb8. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:24.509986 10880 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.511123 10183 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.521734 10259 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.545711 10090 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.565992 10188 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:24.578244 10337 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.581105 10984 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:24.583858 10258 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:24.586455 10984 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:24.608237 10846 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.609885 10337 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:24.643155 10087 catalog_manager.cc:5484] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:24.654083 10263 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
W20240507 08:05:24.667408 10158 consensus_peers.cc:485] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: beb5387c55024337b72bb8aa1ba09a7b. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:24.738855 11003 ts_tablet_manager.cc:902] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:24.740787 11003 tablet_copy_client.cc:316] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:05:24.771384 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet 5d0e1a7b6db34cea8e152b755b52aa96 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:05:24.772085 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet 5d0e1a7b6db34cea8e152b755b52aa96 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-5d0e1a7b6db34cea8e152b755b52aa96
I20240507 08:05:24.782130 10277 tablet_copy_source_session.cc:205] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:24.786666 11003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d0e1a7b6db34cea8e152b755b52aa96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:24.803200 11003 tablet_copy_client.cc:791] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:24.804880 11003 tablet_copy_client.cc:655] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:24.814805 11003 tablet_copy_client.cc:531] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:24.823954 11003 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:24.846843 11007 ts_tablet_manager.cc:902] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:24.855684 11007 tablet_copy_client.cc:316] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:05:24.858194 11003 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:24.859265 11003 tablet_bootstrap.cc:492] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:24.859933 11003 ts_tablet_manager.cc:1367] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.036s user 0.020s sys 0.005s
I20240507 08:05:24.862164 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet 50d914d033cf48c389262dd8b7c7fbce from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:05:24.862807 11003 raft_consensus.cc:348] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:24.863720 11003 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:24.864332 11003 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:24.866443 10968 heartbeater.cc:503] Master 127.8.213.190:37115 was elected leader, sending a full tablet report...
I20240507 08:05:24.870896 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet 50d914d033cf48c389262dd8b7c7fbce from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-50d914d033cf48c389262dd8b7c7fbce
I20240507 08:05:24.907967 10277 tablet_copy_source_session.cc:205] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:24.920049 11007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d914d033cf48c389262dd8b7c7fbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:24.934923 11003 ts_tablet_manager.cc:1398] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.075s user 0.020s sys 0.041s
I20240507 08:05:24.937320 11007 tablet_copy_client.cc:791] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:24.937889 11007 tablet_copy_client.cc:655] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:24.946107 11007 tablet_copy_client.cc:531] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:24.946324 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-5d0e1a7b6db34cea8e152b755b52aa96 received from {username='slave'} at 127.0.0.1:39118
I20240507 08:05:24.949822 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-5d0e1a7b6db34cea8e152b755b52aa96 on tablet 5d0e1a7b6db34cea8e152b755b52aa96 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:24.954942 11003 ts_tablet_manager.cc:902] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:24.961618 11007 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:24.965862 11003 tablet_copy_client.cc:316] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.131:37047
I20240507 08:05:24.977977 10351 tablet_copy_service.cc:140] P 88a26dac37fa4243b8c2b3d47f403488: Received BeginTabletCopySession request for tablet bcbe630c1736458898143fb230a9ffb8 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:47396)
I20240507 08:05:24.978596 10351 tablet_copy_service.cc:161] P 88a26dac37fa4243b8c2b3d47f403488: Beginning new tablet copy session on tablet bcbe630c1736458898143fb230a9ffb8 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:47396: session id = 5d6656a616b747d393ff3de5328d2540-bcbe630c1736458898143fb230a9ffb8
I20240507 08:05:24.999927 10351 tablet_copy_source_session.cc:205] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:25.003564 11003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbe630c1736458898143fb230a9ffb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:25.010793 11007 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:25.011806 11007 tablet_bootstrap.cc:492] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:25.012856 11007 ts_tablet_manager.cc:1367] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.051s user 0.019s sys 0.014s
I20240507 08:05:25.025386 11007 raft_consensus.cc:348] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:25.026423 11007 raft_consensus.cc:729] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:25.027175 11007 consensus_queue.cc:260] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:25.028074 11003 tablet_copy_client.cc:791] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:25.030201 11007 ts_tablet_manager.cc:1398] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20240507 08:05:25.028697 11003 tablet_copy_client.cc:655] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:25.040427 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-50d914d033cf48c389262dd8b7c7fbce received from {username='slave'} at 127.0.0.1:39118
I20240507 08:05:25.040472 11003 tablet_copy_client.cc:531] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:25.040962 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-50d914d033cf48c389262dd8b7c7fbce on tablet 50d914d033cf48c389262dd8b7c7fbce with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:25.051932 11003 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:25.075199 11003 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:25.076020 11003 tablet_bootstrap.cc:492] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:25.076665 11003 ts_tablet_manager.cc:1367] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.003s
I20240507 08:05:25.079100 11003 raft_consensus.cc:348] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:25.079953 11003 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:25.080551 11003 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:25.083093 11003 ts_tablet_manager.cc:1398] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s
I20240507 08:05:25.087410 10351 tablet_copy_service.cc:342] P 88a26dac37fa4243b8c2b3d47f403488: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-bcbe630c1736458898143fb230a9ffb8 received from {username='slave'} at 127.0.0.1:47396
I20240507 08:05:25.087783 10351 tablet_copy_service.cc:433] P 88a26dac37fa4243b8c2b3d47f403488: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-bcbe630c1736458898143fb230a9ffb8 on tablet bcbe630c1736458898143fb230a9ffb8 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:25.099778 10943 raft_consensus.cc:1203] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:25.138814 11003 ts_tablet_manager.cc:902] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:05:25.140580 11003 tablet_copy_client.cc:316] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:05:25.163483 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet beb5387c55024337b72bb8aa1ba09a7b from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:05:25.164002 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet beb5387c55024337b72bb8aa1ba09a7b from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-beb5387c55024337b72bb8aa1ba09a7b
I20240507 08:05:25.177353 10202 tablet_copy_source_session.cc:205] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:25.181778 11003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb5387c55024337b72bb8aa1ba09a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:25.203812 11003 tablet_copy_client.cc:791] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:25.204416 11003 tablet_copy_client.cc:655] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:25.211726 11003 tablet_copy_client.cc:531] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:25.220289 11003 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:25.236955 11003 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:25.237622 11003 tablet_bootstrap.cc:492] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:25.238245 11003 ts_tablet_manager.cc:1367] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.018s user 0.018s sys 0.000s
I20240507 08:05:25.242218 11003 raft_consensus.cc:348] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:25.243098 11003 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:25.243544 11003 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:25.245321 11003 ts_tablet_manager.cc:1398] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240507 08:05:25.246902 10202 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-beb5387c55024337b72bb8aa1ba09a7b received from {username='slave'} at 127.0.0.1:50108
I20240507 08:05:25.247264 10202 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-beb5387c55024337b72bb8aa1ba09a7b on tablet beb5387c55024337b72bb8aa1ba09a7b with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:25.307984 10943 raft_consensus.cc:1203] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:25.410039 10943 raft_consensus.cc:1203] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:25.666357 10983 raft_consensus.cc:1050] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:25.668864 10983 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:25.688758 10943 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:25.690002 10341 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:25.690325 10186 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:25.692610 11016 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:25.696295 11015 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:25.697902 11016 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:25.715097 10942 raft_consensus.cc:1203] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:25.766780 11015 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.775034 10183 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.776453 10337 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.803608 10087 catalog_manager.cc:5484] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:25.807766 10943 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.850260 10258 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:25.861755 10973 raft_consensus.cc:1050] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:25.863541 10973 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:25.871563 10943 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:25.871958 10188 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:25.873772 11016 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:05:25.875136 11025 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:25.882840 10942 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:25.884572 10997 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:25.885404 11015 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.886562 10183 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.900125 10183 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:25.901638 10972 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:25.915968 10258 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:25.924926 10972 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:25.930672 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5d0e1a7b6db34cea8e152b755b52aa96 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:25.962338 10089 catalog_manager.cc:5484] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:25.966194 10973 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.967976 10943 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.984508 10086 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:25.995167 10188 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.017947 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d0e1a7b6db34cea8e152b755b52aa96 with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39708
I20240507 08:05:26.027737 10943 raft_consensus.cc:2932] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.037757 11038 tablet_replica.cc:330] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:05:26.069813 11038 raft_consensus.cc:2229] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:26.070554 11038 raft_consensus.cc:2258] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:26.075557 10261 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.073087 10337 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:26.074420 11015 raft_consensus.cc:1050] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:26.118587 11015 consensus_queue.cc:237] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:26.132735 11038 ts_tablet_manager.cc:1875] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:26.135208 10943 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:26.135620 10183 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:26.137970 11036 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:05:26.138931 10997 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:26.147863 10942 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:26.147982 10337 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:26.149930 10188 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:26.150413 11025 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:26.151301 11016 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:26.158170 11036 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.159159 11026 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:26.165433 10186 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.168295 10943 raft_consensus.cc:2932] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.186877 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bcbe630c1736458898143fb230a9ffb8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:26.197377 10089 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.210369 11038 ts_tablet_manager.cc:1888] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:26.211097 11038 log.cc:1196] T 5d0e1a7b6db34cea8e152b755b52aa96 P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/5d0e1a7b6db34cea8e152b755b52aa96
I20240507 08:05:26.213181 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet 5d0e1a7b6db34cea8e152b755b52aa96 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:26.211619 11025 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.223884 10337 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.229614 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbe630c1736458898143fb230a9ffb8 with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42910
I20240507 08:05:26.248582 10942 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.250156 10188 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.273418 11043 tablet_replica.cc:330] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:05:26.274698 11043 raft_consensus.cc:2229] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:26.274104 10087 catalog_manager.cc:5484] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.275475 11043 raft_consensus.cc:2258] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:26.279579 11043 ts_tablet_manager.cc:1875] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:26.300968 11043 ts_tablet_manager.cc:1888] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:26.301455 11043 log.cc:1196] T bcbe630c1736458898143fb230a9ffb8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/bcbe630c1736458898143fb230a9ffb8
I20240507 08:05:26.304422 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet bcbe630c1736458898143fb230a9ffb8 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:26.304016 10261 consensus_queue.cc:237] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:26.311084 10984 raft_consensus.cc:1050] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:26.313644 10984 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:26.331771 10339 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:26.333871 11025 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:26.335987 10942 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:26.339529 11025 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:26.345120 11016 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.355080 10339 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.356705 10337 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:26.358772 10942 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:26.359485 11024 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:26.363058 10846 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:26.373080 10258 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:26.387773 10984 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:26.390710 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 50d914d033cf48c389262dd8b7c7fbce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:26.393637 10942 raft_consensus.cc:2932] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.394577 11045 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.407438 10341 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.414919 10089 catalog_manager.cc:5484] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:26.417455 10942 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.429958 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet 50d914d033cf48c389262dd8b7c7fbce with delete_type TABLET_DATA_TOMBSTONED (TS 59382e6647c549d89d83c70233c0667e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55810
I20240507 08:05:26.430181 10087 catalog_manager.cc:5484] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:26.440647 10263 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.443634 11049 tablet_replica.cc:330] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: stopping tablet replica
I20240507 08:05:26.444753 11049 raft_consensus.cc:2229] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:26.452256 11049 raft_consensus.cc:2258] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:26.467556 11049 ts_tablet_manager.cc:1875] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:26.497032 11049 ts_tablet_manager.cc:1888] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:26.497531 11049 log.cc:1196] T 50d914d033cf48c389262dd8b7c7fbce P 59382e6647c549d89d83c70233c0667e: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/wals/50d914d033cf48c389262dd8b7c7fbce
I20240507 08:05:26.500403 10076 catalog_manager.cc:4830] TS 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): tablet 50d914d033cf48c389262dd8b7c7fbce (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:26.499408 10186 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:26.508157 10341 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:26.509956 10984 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:26.512543 10942 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:26.519323 11024 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:26.519048 11045 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.523234 10942 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.526387 10341 raft_consensus.cc:2932] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:26.532100 10076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet beb5387c55024337b72bb8aa1ba09a7b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:26.536608 10089 catalog_manager.cc:5484] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:26.548791 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet beb5387c55024337b72bb8aa1ba09a7b with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42910
I20240507 08:05:26.549755 11043 tablet_replica.cc:330] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:05:26.550751 11043 raft_consensus.cc:2229] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:26.551513 11043 raft_consensus.cc:2258] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:26.555387 11043 ts_tablet_manager.cc:1875] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:26.578984 11043 ts_tablet_manager.cc:1888] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:26.579422 11043 log.cc:1196] T beb5387c55024337b72bb8aa1ba09a7b P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/beb5387c55024337b72bb8aa1ba09a7b
I20240507 08:05:26.580819 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet beb5387c55024337b72bb8aa1ba09a7b (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:27.604342 10186 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:27.610353 10263 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.610493 10341 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240507 08:05:27.610823 10158 consensus_peers.cc:485] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a168a24022a4678bd74fa028a765700. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:27.611599 10984 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:27.612360 11024 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:27.618953 11024 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.620536 10341 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.621855 10263 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.638209 10087 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.647949 10263 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
W20240507 08:05:27.657915 10233 consensus_peers.cc:485] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f4b60e5e6294639a468641f215329a5. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:27.657460 10188 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.659687 10337 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.660095 11052 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:27.661984 11052 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:27.686215 11052 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.691140 10337 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.707337 10186 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:27.719527 10090 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:27.724400 10188 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.738692 10337 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.740043 10984 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240507 08:05:27.740862 10158 consensus_peers.cc:485] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6ac20a6d50a4700a577dfc18d71ceb2. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:27.741469 10263 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.744755 10984 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:27.747040 11024 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.748484 10337 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.752750 10263 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.759491 10258 consensus_queue.cc:237] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:27.764106 10090 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.780474 10337 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.781895 10188 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:27.784286 11067 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:27.786583 11052 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240507 08:05:27.793937 10233 consensus_peers.cc:485] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e6f917a0bed4f6289c1abc13f6260b9. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:27.793699 11052 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.794332 10337 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.814970 10089 catalog_manager.cc:5484] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:27.823971 10186 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:28.072153 11070 ts_tablet_manager.cc:902] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:28.074942 11070 tablet_copy_client.cc:316] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:05:28.076468 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet 1f4b60e5e6294639a468641f215329a5 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:05:28.076998 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet 1f4b60e5e6294639a468641f215329a5 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-1f4b60e5e6294639a468641f215329a5
I20240507 08:05:28.084205 10277 tablet_copy_source_session.cc:205] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:28.086741 11070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f4b60e5e6294639a468641f215329a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:28.101656 11070 tablet_copy_client.cc:791] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:28.102216 11070 tablet_copy_client.cc:655] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:28.109314 11070 tablet_copy_client.cc:531] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:28.116506 11070 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:28.133939 11070 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:28.134761 11070 tablet_bootstrap.cc:492] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:28.135445 11070 ts_tablet_manager.cc:1367] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.005s
I20240507 08:05:28.138240 11070 raft_consensus.cc:348] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.138981 11070 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:28.139453 11070 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.142148 11070 ts_tablet_manager.cc:1398] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20240507 08:05:28.145022 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-1f4b60e5e6294639a468641f215329a5 received from {username='slave'} at 127.0.0.1:39118
I20240507 08:05:28.145412 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-1f4b60e5e6294639a468641f215329a5 on tablet 1f4b60e5e6294639a468641f215329a5 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:28.151199 11074 ts_tablet_manager.cc:902] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:05:28.152999 11074 tablet_copy_client.cc:316] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:05:28.154402 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet 5a168a24022a4678bd74fa028a765700 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:05:28.154912 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet 5a168a24022a4678bd74fa028a765700 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-5a168a24022a4678bd74fa028a765700
I20240507 08:05:28.158200 11070 ts_tablet_manager.cc:902] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:05:28.160733 10202 tablet_copy_source_session.cc:205] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:28.161741 11070 tablet_copy_client.cc:316] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:05:28.163208 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet c6ac20a6d50a4700a577dfc18d71ceb2 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:05:28.163758 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet c6ac20a6d50a4700a577dfc18d71ceb2 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-c6ac20a6d50a4700a577dfc18d71ceb2
I20240507 08:05:28.164335 11074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a168a24022a4678bd74fa028a765700. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:28.173818 10202 tablet_copy_source_session.cc:205] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:28.176891 11070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ac20a6d50a4700a577dfc18d71ceb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:28.187072 11074 tablet_copy_client.cc:791] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:28.187680 11074 tablet_copy_client.cc:655] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:28.191504 11074 tablet_copy_client.cc:531] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:28.203177 11074 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:28.209316 11070 tablet_copy_client.cc:791] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:28.209913 11070 tablet_copy_client.cc:655] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:28.214355 11070 tablet_copy_client.cc:531] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:28.224543 11074 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:28.225677 11074 tablet_bootstrap.cc:492] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:28.226612 11070 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:28.228164 11074 ts_tablet_manager.cc:1367] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.005s
I20240507 08:05:28.237529 11074 raft_consensus.cc:348] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.239565 11074 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:28.240295 11074 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.243300 11074 ts_tablet_manager.cc:1398] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20240507 08:05:28.247268 10202 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-5a168a24022a4678bd74fa028a765700 received from {username='slave'} at 127.0.0.1:50108
I20240507 08:05:28.247668 10202 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-5a168a24022a4678bd74fa028a765700 on tablet 5a168a24022a4678bd74fa028a765700 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:28.251148 11070 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:28.252054 11070 tablet_bootstrap.cc:492] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:28.252696 11070 ts_tablet_manager.cc:1367] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.026s user 0.012s sys 0.010s
I20240507 08:05:28.255203 11070 raft_consensus.cc:348] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.256037 11070 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:28.256621 11070 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.258622 11070 ts_tablet_manager.cc:1398] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20240507 08:05:28.260324 10202 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-c6ac20a6d50a4700a577dfc18d71ceb2 received from {username='slave'} at 127.0.0.1:50108
I20240507 08:05:28.260751 10202 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-c6ac20a6d50a4700a577dfc18d71ceb2 on tablet c6ac20a6d50a4700a577dfc18d71ceb2 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:28.342142 11070 ts_tablet_manager.cc:902] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:28.343607 11070 tablet_copy_client.cc:316] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:05:28.344951 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet 7e6f917a0bed4f6289c1abc13f6260b9 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:05:28.345355 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet 7e6f917a0bed4f6289c1abc13f6260b9 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-7e6f917a0bed4f6289c1abc13f6260b9
I20240507 08:05:28.352077 10277 tablet_copy_source_session.cc:205] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:28.354895 11070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e6f917a0bed4f6289c1abc13f6260b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:28.368556 11070 tablet_copy_client.cc:791] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:28.369135 11070 tablet_copy_client.cc:655] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:28.372594 11070 tablet_copy_client.cc:531] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:28.379776 11070 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:28.394941 11070 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:28.395684 11070 tablet_bootstrap.cc:492] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:28.396162 11070 ts_tablet_manager.cc:1367] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.001s
I20240507 08:05:28.398308 11070 raft_consensus.cc:348] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.398967 11070 raft_consensus.cc:729] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:28.399426 11070 consensus_queue.cc:260] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:28.401149 11070 ts_tablet_manager.cc:1398] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240507 08:05:28.406700 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-7e6f917a0bed4f6289c1abc13f6260b9 received from {username='slave'} at 127.0.0.1:39118
I20240507 08:05:28.407109 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-7e6f917a0bed4f6289c1abc13f6260b9 on tablet 7e6f917a0bed4f6289c1abc13f6260b9 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:28.679601 10942 raft_consensus.cc:1203] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:28.710947 10942 raft_consensus.cc:1203] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:28.719352 10942 raft_consensus.cc:1203] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:28.926151 10942 raft_consensus.cc:1203] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:29.156378 11067 raft_consensus.cc:1050] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:29.159335 11067 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:29.165623 10942 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:29.165805 10337 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:29.167125 11052 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:29.168011 11025 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:29.170742 10186 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:29.172809 11084 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:29.184798 11085 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.186374 10337 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.195153 10942 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.207230 10087 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:29.205001 10188 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.217082 11082 raft_consensus.cc:1050] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:29.219583 11082 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:29.236347 11050 raft_consensus.cc:1050] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:29.238845 11050 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:29.244454 10261 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:29.251982 10942 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:29.254328 10337 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:29.254540 11025 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:05:29.255754 11067 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:29.258613 10337 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:29.263356 11067 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.266943 10333 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.274721 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1f4b60e5e6294639a468641f215329a5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:29.275038 10942 raft_consensus.cc:2932] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:29.295384 11093 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20240507 08:05:29.976591 10155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.010s sys 0.011s
W20240507 08:05:29.978026 10155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.696s user 0.010s sys 0.014s
W20240507 08:05:29.980170 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 698733us
I20240507 08:05:30.022081 10258 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.034984 11101 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:30.054600 11109 raft_consensus.cc:1050] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:05:30.057296 10156 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.005s sys 0.006s
W20240507 08:05:30.057773 10156 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.773s user 0.007s sys 0.006s
I20240507 08:05:30.058148 11109 consensus_queue.cc:237] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:30.105191 10263 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.107192 11121 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:30.119026 10942 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.119385 10943 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.192646 11108 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:30.194439 11102 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:30.258649 11124 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.259260 11124 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.258673 11093 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.263294 11093 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.265156 11093 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.267916 10941 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.262157 10086 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:05:30.271209 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 58456.5us
I20240507 08:05:30.261368 11124 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.286208 11093 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:30.286764 11093 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:30.288846 11093 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.292901 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.309239 11100 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:30.310168 10333 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.314883 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.333056 11124 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.333848 11124 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.334882 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.336247 11124 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.338671 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.340067 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.346668 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.354563 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.367234 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.368211 11105 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:30.368778 11105 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.369647 10192 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.367235 11128 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:30.370663 11128 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:30.373139 11121 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.373764 11121 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.373237 11128 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:30.375651 11121 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.375651 11105 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:30.434093 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 52853.2us
I20240507 08:05:30.435695 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.445858 11124 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.446642 11127 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:30.450984 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 103070us
I20240507 08:05:30.459165 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.436841 11100 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.511349 11100 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.508627 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.508847 10335 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:30.447326 11127 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.520213 11127 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.506295 11111 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:30.509975 11130 log.cc:924] Time spent T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:05:30.446691 11124 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.394065 11109 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.112s
W20240507 08:05:30.517961 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 57553.5us
W20240507 08:05:30.522697 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52577us
I20240507 08:05:30.525725 10156 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.526901 10156 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.528199 11121 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:30.530467 11121 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.530625 11102 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.531178 11102 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.531682 11102 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.532775 11124 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.533659 11131 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.534194 11131 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.537421 11131 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.544724 11102 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.546531 11103 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:30.546943 11101 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:30.547452 11101 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.523438 11111 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.551476 11111 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.553440 11100 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.557682 11101 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:30.561132 11102 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.565402 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.574579 11112 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:30.575075 11112 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:30.576756 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.577450 11112 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:30.582101 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.585465 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.587129 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:30.588912 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.589504 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:30.591539 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.593953 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.594463 10267 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.598184 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:30.607317 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:30.611382 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 83574.7us
I20240507 08:05:30.615386 11111 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.619516 10156 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.620895 10156 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.624047 11131 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.620502 10232 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:30.623131 11107 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.629868 11104 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.635322 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.636420 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.640750 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.641363 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.640825 10337 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.643662 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.643975 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.641384 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.632203 10942 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.648986 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:30.654116 10943 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.666954 10941 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.680702 10261 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.685779 11101 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.695915 10306 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.703092 10306 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.706833 11106 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.707562 11124 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:30.708288 11124 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:30.713708 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:30.724812 11105 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.727645 10306 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.739245 10156 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.734207 11105 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.744848 10156 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.743434 11127 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.749619 11122 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.757531 10156 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.761902 11127 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.760905 10156 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.770596 10186 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.787642 10156 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.789369 10156 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.792610 10156 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.793318 11101 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.791253 11121 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.793365 11093 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.788970 11122 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.799537 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f4b60e5e6294639a468641f215329a5 with delete_type TABLET_DATA_TOMBSTONED (TS 59382e6647c549d89d83c70233c0667e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55810
I20240507 08:05:30.787397 10089 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:30.806465 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:30.758985 11124 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:30.811457 10156 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:30.800603 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:30.819445 10335 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.816891 11121 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:30.837147 10087 catalog_manager.cc:5484] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.848695 11148 tablet_replica.cc:330] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: stopping tablet replica
I20240507 08:05:30.847625 10332 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.857833 11148 raft_consensus.cc:2229] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:30.858443 11148 raft_consensus.cc:2258] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:30.860816 11148 ts_tablet_manager.cc:1875] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:30.863634 10090 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.867645 10186 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:30.878837 11148 ts_tablet_manager.cc:1888] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:30.879292 11148 log.cc:1196] T 1f4b60e5e6294639a468641f215329a5 P 59382e6647c549d89d83c70233c0667e: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/wals/1f4b60e5e6294639a468641f215329a5
I20240507 08:05:30.886682 10262 consensus_queue.cc:237] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:30.891175 10259 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:30.896862 11093 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:30.899559 10942 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:30.902487 10076 catalog_manager.cc:4830] TS 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): tablet 1f4b60e5e6294639a468641f215329a5 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:30.905506 11121 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:05:30.905531 10186 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:30.907239 10332 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:30.907647 11137 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.909431 11104 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:30.911832 10942 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:30.914922 10260 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:30.916060 11116 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.912837 10262 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.920161 11093 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:30.920217 11100 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:05:30.921834 10076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a168a24022a4678bd74fa028a765700 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:30.928875 10942 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.931674 10943 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:30.933508 11121 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:30.932564 10941 raft_consensus.cc:2932] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.938241 10090 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.945045 10940 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.939293 11093 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.956935 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a168a24022a4678bd74fa028a765700 with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39708
I20240507 08:05:30.961689 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e6f917a0bed4f6289c1abc13f6260b9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:30.963521 10088 catalog_manager.cc:5484] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.965890 10076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6ac20a6d50a4700a577dfc18d71ceb2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:30.968223 10262 raft_consensus.cc:2932] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.977027 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e6f917a0bed4f6289c1abc13f6260b9 with delete_type TABLET_DATA_TOMBSTONED (TS 59382e6647c549d89d83c70233c0667e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55810
I20240507 08:05:30.918174 10335 raft_consensus.cc:2932] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:30.978060 11148 tablet_replica.cc:330] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: stopping tablet replica
I20240507 08:05:30.979046 11148 raft_consensus.cc:2229] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:30.979748 11148 raft_consensus.cc:2258] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:30.983181 11148 ts_tablet_manager.cc:1875] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:30.984855 11149 tablet_replica.cc:330] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:05:30.989496 11149 raft_consensus.cc:2229] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:30.990293 11149 raft_consensus.cc:2258] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:31.000229 11149 ts_tablet_manager.cc:1875] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:31.005662 11148 ts_tablet_manager.cc:1888] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:31.006220 11148 log.cc:1196] T 7e6f917a0bed4f6289c1abc13f6260b9 P 59382e6647c549d89d83c70233c0667e: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/wals/7e6f917a0bed4f6289c1abc13f6260b9
I20240507 08:05:31.007701 10076 catalog_manager.cc:4830] TS 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): tablet 7e6f917a0bed4f6289c1abc13f6260b9 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:31.018246 11149 ts_tablet_manager.cc:1888] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:31.018827 11149 log.cc:1196] T 5a168a24022a4678bd74fa028a765700 P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/5a168a24022a4678bd74fa028a765700
I20240507 08:05:31.019757 10088 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 1 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:31.021144 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet 5a168a24022a4678bd74fa028a765700 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:31.039244 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ac20a6d50a4700a577dfc18d71ceb2 with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39708
I20240507 08:05:31.040108 11149 tablet_replica.cc:330] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:05:31.041126 11149 raft_consensus.cc:2229] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:31.041854 11149 raft_consensus.cc:2258] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:31.046883 11149 ts_tablet_manager.cc:1875] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:31.059049 11149 ts_tablet_manager.cc:1888] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:31.059345 11149 log.cc:1196] T c6ac20a6d50a4700a577dfc18d71ceb2 P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/c6ac20a6d50a4700a577dfc18d71ceb2
I20240507 08:05:31.060480 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet c6ac20a6d50a4700a577dfc18d71ceb2 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
W20240507 08:05:31.145004 10362 debug-util.cc:398] Leaking SignalData structure 0x7b0800267c60 after lost signal to thread 9049
W20240507 08:05:31.145813 10362 debug-util.cc:398] Leaking SignalData structure 0x7b0800281480 after lost signal to thread 10124
W20240507 08:05:31.146744 10362 debug-util.cc:398] Leaking SignalData structure 0x7b080016c380 after lost signal to thread 10216
W20240507 08:05:31.147627 10362 debug-util.cc:398] Leaking SignalData structure 0x7b0800267c40 after lost signal to thread 10291
W20240507 08:05:31.148558 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08002c8d40 after lost signal to thread 10365
W20240507 08:05:31.149519 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08002fbcc0 after lost signal to thread 10967
I20240507 08:05:33.062278 11153 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.063679 11153 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.067627 11152 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.068467 11152 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.071082 11152 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:33.071851 10121 debug-util.cc:398] Leaking SignalData structure 0x7b0800064360 after lost signal to thread 9049
I20240507 08:05:33.073417 11153 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.126600 11112 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.127306 11112 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.129916 11112 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:33.145960 11154 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.147372 11154 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.146624 11156 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.148821 11156 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.150460 11116 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.151199 11158 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.151796 11121 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.151150 11157 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.158751 11121 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.159675 11157 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.161232 11121 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.151134 11116 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.161557 11157 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.152719 11156 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.154785 11154 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.168067 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.170388 11157 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.170797 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.171006 11157 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.165051 11116 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.173151 11157 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.174301 11159 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.161983 11158 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.175873 11154 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.174870 11159 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.176482 11154 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.178547 11121 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.182346 11154 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.186517 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.186972 11121 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.190232 11121 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.184762 11162 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.200783 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.201741 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.200572 11160 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.202865 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.202610 11160 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.192195 11162 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.205911 11160 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.206677 11162 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.192149 11158 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.207093 11162 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.229614 11162 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.232658 11162 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.235548 11158 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.236157 11158 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.236474 11137 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.237131 11137 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.238221 11158 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.239048 11137 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.239758 11158 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.240682 11104 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.258210 11158 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.259239 11168 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.259757 11168 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.260403 11158 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.263576 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.264572 10189 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:33.264750 11104 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.267469 11104 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.269928 10908 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:33.271217 11137 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.272033 11137 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.281304 11159 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.275660 11166 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.282085 11164 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.282317 11166 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.288697 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.289685 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.289727 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.276259 11137 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.275918 11151 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.291992 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.291913 11151 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.293506 10332 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:33.290035 11166 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.295744 11151 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.273744 11167 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.303081 11167 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.303526 11167 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.284976 11159 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.306041 11159 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.296393 11100 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.309113 11100 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.297108 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:33.300072 10121 debug-util.cc:398] Leaking SignalData structure 0x7b0800064a40 after lost signal to thread 10124
I20240507 08:05:33.314815 11093 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.315194 11159 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:33.316685 11100 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.314934 11152 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.320684 11152 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.321246 11168 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.321974 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.322471 11168 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.323161 11168 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.315642 11093 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:33.326830 11157 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.327435 11157 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.328678 11093 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.335366 11161 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:33.335752 10121 debug-util.cc:398] Leaking SignalData structure 0x7b08001fb500 after lost signal to thread 10216
I20240507 08:05:33.322727 11152 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.336000 11161 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.282899 11164 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.329447 11157 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:33.346648 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 127405us
I20240507 08:05:33.346920 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.347885 11116 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.353035 11104 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.348462 11116 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.353711 11104 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.355679 11104 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.355794 11116 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.339246 11100 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:33.357087 10121 debug-util.cc:398] Leaking SignalData structure 0x7b080031e9a0 after lost signal to thread 10291
W20240507 08:05:33.365844 10121 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdf40 after lost signal to thread 10365
I20240507 08:05:33.366678 11121 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:33.366911 10121 debug-util.cc:398] Leaking SignalData structure 0x7b08003021a0 after lost signal to thread 10967
I20240507 08:05:33.367298 11121 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.369880 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.369346 11121 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.373646 11171 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.373750 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.374320 11171 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.376417 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.376641 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.377348 11153 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.383185 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.383383 11153 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.383783 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.385040 11162 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.374809 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.392639 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:33.401345 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.383266 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.405154 11170 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.405520 10185 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.398447 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.398144 11175 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.399741 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:33.406888 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 67600.4us
I20240507 08:05:33.385360 11156 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.409785 11116 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.398483 11154 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.387125 11162 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.396694 11161 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.411015 10336 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:33.416716 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.385516 11153 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.389824 11158 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.423344 11158 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.424005 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.417086 11156 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.397356 11168 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.393162 11152 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.421867 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.422822 11162 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.425697 11152 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.390707 11121 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.426254 11156 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.394716 11093 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.429755 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.429977 11158 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.413511 11171 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.405740 11170 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.414608 11116 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.392477 11167 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.434558 11165 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.398802 11100 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.436826 11162 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.436990 11093 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.439822 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.435113 11165 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.438958 11112 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.435592 11116 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.426852 11121 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.438108 11162 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.445044 11165 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.442725 11093 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.445890 11162 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.438341 11171 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.447047 11171 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.439054 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.410658 11175 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.440661 11152 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.440960 11112 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.418532 11154 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.433944 11160 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.433403 11161 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.437423 11168 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.449354 11171 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.457094 11165 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.457517 11175 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.457628 11165 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.458190 11161 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.459656 11165 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.460023 11161 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.460767 11112 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.461383 11121 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.454932 11160 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.461890 11168 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.463168 11160 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.463984 11168 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.467427 10940 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.467823 11154 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.472198 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:33.472620 11170 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.472887 10940 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.474463 11167 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.474996 11100 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.475582 11164 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.478374 11121 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.478912 11100 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.479099 11121 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.479447 11100 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.479977 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:33.483273 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52402.9us
I20240507 08:05:33.484422 11121 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.484804 11160 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.483904 11152 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.484980 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.486001 11160 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.486083 11152 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.486290 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.486932 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.487763 11116 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.488070 11152 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.488531 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.489719 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.490607 11116 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.491222 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.491876 10188 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.491222 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.492192 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.492841 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.498198 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.500890 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.501137 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.504318 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.504662 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.505925 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.506078 11175 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.506629 11175 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.507432 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.507932 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.503549 11180 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.508522 11175 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.508549 10262 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.508915 11180 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.509469 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.511413 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.505941 11160 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.512787 11165 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.512918 11100 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.513130 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.513306 11165 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.514668 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.511577 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.515210 11165 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.515673 11179 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.519131 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.519404 11165 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.519549 11170 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:33.519872 11165 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.520640 11170 raft_consensus.cc:3032] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.521042 11170 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:33.522462 11165 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.523953 10306 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:33.523885 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.525528 11177 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.534448 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.534659 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.534698 11177 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.535274 11177 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.535943 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.536805 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:33.536185 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.524538 11165 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.526399 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.526679 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.539252 11165 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.516186 11179 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.541265 11165 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.526196 11170 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.528352 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.542086 10261 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.542629 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.530480 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:33.530725 11180 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:33.525691 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.530784 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.547335 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:33.532645 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.548116 11177 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.533457 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.550120 11177 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.541342 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.531296 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.529567 10307 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:33.557621 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.558076 10307 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:33.532526 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.545457 11171 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.532018 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.558863 11171 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.531855 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:33.559259 11171 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.559666 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.561868 11178 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.562278 11178 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.562623 11178 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.563925 11112 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.564298 11112 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.564646 11112 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.561189 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.566594 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.538666 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.558605 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.551810 11170 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.546334 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.567934 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.553961 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.569142 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.569324 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.566599 11171 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.554637 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.571940 11172 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:33.554113 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.555778 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.572535 11172 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.573107 11171 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.555806 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.558132 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.573647 11171 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.566699 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.570312 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.554931 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.574965 11172 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.576174 11180 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:33.576668 11180 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.552820 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.577847 10260 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.577931 11165 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.578408 11165 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.578836 11180 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.546720 11179 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.580289 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.581821 11165 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.582788 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.582692 11178 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.584465 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.588447 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.589360 10260 raft_consensus.cc:2379] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:33.590435 11112 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.591043 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.592746 11112 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:33.593317 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:33.593887 10940 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.596081 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.596259 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.596552 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.600523 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.600593 10940 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:33.600811 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:33.601559 10307 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:33.602407 11170 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:33.603693 11170 raft_consensus.cc:686] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:33.604593 11170 consensus_queue.cc:237] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.606952 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.607733 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.608183 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:33.606344 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.609637 11178 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:33.610095 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.612125 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.612433 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.614073 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.615443 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.616230 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.616528 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.616815 10261 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.616966 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.617193 10260 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.618448 10337 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:33.619700 10232 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.620088 10907 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.621168 11167 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:33.621665 11167 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:33.622071 11179 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.622524 11167 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.624475 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.624680 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.625125 10260 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.625320 10261 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.625304 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.626055 10258 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.626156 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.626735 10259 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.627593 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.628194 10258 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.630019 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.630306 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.629385 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.631740 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.631884 10258 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:33.638922 10232 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.639271 10940 raft_consensus.cc:1226] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.640065 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.641314 10232 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.641345 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:33.642271 11100 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.642827 10906 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.643093 10304 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.643402 11160 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.643728 11171 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.642838 11100 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.644179 11171 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.644564 11171 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.644613 11167 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.644789 10232 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.645793 11100 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.645931 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.646832 10232 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.636518 10090 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 2 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:33.647558 11100 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.651203 10232 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.651441 11171 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.651983 11161 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.652788 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.654516 11100 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.655318 11100 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.655364 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.653277 10304 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.656509 11112 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.656193 11100 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.658838 11100 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:33.660923 11171 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.662351 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.664665 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.665602 11161 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.666910 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.667107 11112 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.667542 11112 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.667927 11112 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.668840 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.666927 10232 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.670982 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.674080 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.676070 11112 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.683499 11112 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.685010 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.680330 10304 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.686611 11161 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.687294 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.688427 10304 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.689404 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.690074 11164 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.697834 11164 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.698222 11164 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.698388 10261 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.699823 11161 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.690495 11177 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.690881 11171 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.693852 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.695142 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.697121 10232 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.709173 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.710104 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.710812 11154 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.711190 11154 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.711300 11156 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.711583 11154 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.711647 11156 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.710119 10232 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.710636 11177 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.712603 11161 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.712641 10304 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.713922 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.713500 10232 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.715107 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.713150 11100 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.712064 11156 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.718042 10304 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.719069 10304 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.719094 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.719813 11100 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.721081 10304 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.729856 11171 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.724368 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.730760 11112 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.731132 10304 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.731303 11171 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.731739 11171 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.732306 10304 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.721822 11177 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.732972 11177 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.731128 11112 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.720347 11170 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.733456 11112 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.733717 11168 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.734375 11168 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.733891 11177 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.739180 11154 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.739369 11164 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.739686 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.741182 11154 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.742324 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.742740 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.743808 11112 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.745400 11112 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.728317 11161 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.746536 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.747417 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.747635 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.748008 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.748662 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.748862 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.749138 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.742902 11164 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.721554 10232 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.752228 11161 consensus_queue.cc:1046] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:33.733418 10304 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.752461 11116 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.753458 11100 raft_consensus.cc:3032] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.754133 11100 raft_consensus.cc:729] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:33.733687 11170 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.755019 11100 consensus_queue.cc:260] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:33.755591 11170 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.755618 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.756271 11100 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.763566 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.763873 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:33.767391 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:33.769079 11171 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.764917 10088 catalog_manager.cc:5484] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 2 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:33.770958 11171 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.767288 11156 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.773236 11156 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.753747 11164 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.775492 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.775945 10337 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.776542 11172 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.774844 11164 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.777951 11164 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.777026 10339 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.777400 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:33.779382 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 57901.6us
I20240507 08:05:33.780615 10232 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.780120 11178 consensus_queue.cc:1046] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:33.781904 11172 raft_consensus.cc:3032] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.782235 11172 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.783082 11172 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.784224 11172 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.784399 11160 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.782105 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.787345 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.789268 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.789425 11121 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.790010 11178 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.785737 10232 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.786413 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.791507 10232 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.792603 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.792615 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.793751 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.793960 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.794777 11161 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.793572 10232 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.798321 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:33.799583 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.800192 11161 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.802273 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.802884 11164 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.803440 10337 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:33.804917 11164 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.805825 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.796473 11116 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.807507 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.807049 10906 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.808116 11153 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:33.808400 11162 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.808677 11167 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.809626 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.810547 10155 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.810629 11158 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.811982 10232 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.812688 11158 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.812831 11153 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.812688 10155 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.798022 11100 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:33.814491 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.814689 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.815163 10339 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.809274 11178 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.813882 11137 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.817410 11178 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.817448 10304 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.818138 11137 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.819362 10339 raft_consensus.cc:1226] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.816165 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.813747 11151 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:33.819561 11112 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.826262 11121 consensus_queue.cc:1046] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240507 08:05:33.826649 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.824952 11151 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:33.827373 11151 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.827340 11121 raft_consensus.cc:3032] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.825363 10155 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.829294 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.830897 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.831614 10232 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.833086 10232 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.834625 10155 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.834605 10232 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.836267 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:33.818331 11178 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.832724 11121 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.828187 11171 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.836006 11158 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.838711 10304 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.835285 11159 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.837764 11116 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.839392 11173 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.838035 11100 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.844918 10304 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.845963 11178 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.846091 11168 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.842168 11177 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.850934 11177 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.852180 11155 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.839840 11170 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.835052 11153 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.818751 11137 consensus_queue.cc:1046] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240507 08:05:33.842972 10155 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.837936 11161 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.853385 10304 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.838536 10232 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.874512 11116 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.875675 10232 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.876700 11116 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.877769 10232 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:33.858985 11121 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.882895 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:33.883404 11121 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:33.886022 10307 consensus_peers.cc:485] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9adf1535c974b2abf910f3c1c537627. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:33.883996 11128 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240507 08:05:33.887233 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 44333.3us
I20240507 08:05:33.888738 10155 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.887414 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:33.885100 10258 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:33.844748 11171 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.886962 11116 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.889616 11137 raft_consensus.cc:3032] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.891081 11137 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.888006 11153 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.896322 11170 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:33.897207 10155 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.899053 10155 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.903537 10188 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.904510 11174 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.894255 11137 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.905598 11178 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.905606 11137 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.906158 10155 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.907418 10155 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.908430 11172 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.909529 10155 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.910840 10337 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.914534 10155 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.914712 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:33.914927 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 77512.7us
I20240507 08:05:33.898905 10336 raft_consensus.cc:1226] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.917260 11152 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.917740 11173 consensus_queue.cc:1046] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:33.917222 10306 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.914093 11178 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.902494 11153 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.920753 11189 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.918802 11173 raft_consensus.cc:3032] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.923826 10232 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:33.925101 11116 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.927052 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.930303 11158 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:33.929561 10306 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:33.919180 11174 consensus_queue.cc:1046] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20240507 08:05:33.932886 11174 raft_consensus.cc:3032] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.933208 11174 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.931725 10188 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:33.933822 11189 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.913655 11172 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:33.934415 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:33.924726 11173 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.939513 10337 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.944254 11189 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240507 08:05:33.967166 11161 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:33.971089 11160 raft_consensus.cc:3032] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.971539 11160 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:33.972368 11160 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.973369 10336 raft_consensus.cc:1226] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.937546 11173 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.975313 11173 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.975117 11160 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.976030 11152 consensus_queue.cc:1046] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:33.933984 11174 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.978833 11174 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:33.978963 11093 consensus_queue.cc:1159] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20240507 08:05:33.981303 11178 raft_consensus.cc:3032] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.982800 11178 raft_consensus.cc:729] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.984094 11178 consensus_queue.cc:260] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:33.985510 11170 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:33.986006 11178 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:33.992338 10336 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.996115 10337 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:33.995442 11152 consensus_queue.cc:1046] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240507 08:05:33.997709 11152 raft_consensus.cc:3032] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:33.998144 11152 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:33.997658 11093 consensus_queue.cc:1046] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240507 08:05:33.998910 11152 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.000329 11093 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:34.000655 11093 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:34.000717 11152 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.001482 11093 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.003698 11093 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.020713 10260 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:34.022195 11151 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.023479 11155 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:34.024370 11151 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:34.034801 10337 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:34.037426 11165 consensus_queue.cc:1046] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:34.039487 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:34.039999 11180 raft_consensus.cc:3032] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:34.040414 11180 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:34.041186 11180 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.037295 11188 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.042757 10337 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:34.042980 11180 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.046855 11121 consensus_queue.cc:1046] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.041471 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:34.050108 10306 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:34.052475 11156 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:34.044572 10183 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.053200 11137 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:34.053536 11137 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:34.055325 11137 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.056627 11137 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.063694 10259 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.066905 10190 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.089565 10189 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:34.089617 10088 catalog_manager.cc:5484] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.094406 10190 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:34.097235 11167 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:34.099022 10942 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:34.103615 11155 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:34.104705 10306 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:34.111135 11151 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:34.117036 10337 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.113315 11151 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:34.127642 10336 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:34.154268 11167 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:34.153457 11151 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.173791 10335 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:34.176054 10339 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
W20240507 08:05:34.176214 10307 consensus_peers.cc:485] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f7d05e73be847e99048092c9570faed. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:34.178217 11152 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.179332 11152 consensus_queue.cc:1046] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.180225 11152 raft_consensus.cc:3032] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:34.180438 11152 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:34.180860 10189 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:34.178580 10259 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:34.181720 11178 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:34.182063 11178 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:34.182821 11178 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.183085 11190 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.184110 11178 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.181047 11152 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:34.186789 11152 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.190428 11190 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:34.202180 11155 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.216037 10259 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.220193 10183 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.225843 10090 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 2 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:34.236989 10088 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.276400 10260 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
W20240507 08:05:34.298763 10233 consensus_peers.cc:485] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0f47233c3be4fe99e097ade2aaec1e8. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:34.300199 10183 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:34.302014 10336 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:34.302119 11180 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.303578 11165 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.313966 11180 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.318971 10336 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.324553 10190 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.330411 10337 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.334960 10090 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:34.353830 10259 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:05:34.355690 11155 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.356142 10189 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240507 08:05:34.356657 10307 consensus_peers.cc:485] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84da12b367b046b7bcfb08aaa6e1c97b. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:05:34.358369 11155 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.371054 11155 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.374576 10189 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.378845 10259 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:05:34.400159 10086 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 2, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:34.427115 11212 ts_tablet_manager.cc:902] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:34.439395 11212 tablet_copy_client.cc:316] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.131:37047
I20240507 08:05:34.441821 10351 tablet_copy_service.cc:140] P 88a26dac37fa4243b8c2b3d47f403488: Received BeginTabletCopySession request for tablet e9adf1535c974b2abf910f3c1c537627 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:47396)
I20240507 08:05:34.442523 10351 tablet_copy_service.cc:161] P 88a26dac37fa4243b8c2b3d47f403488: Beginning new tablet copy session on tablet e9adf1535c974b2abf910f3c1c537627 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:47396: session id = 5d6656a616b747d393ff3de5328d2540-e9adf1535c974b2abf910f3c1c537627
I20240507 08:05:34.456825 10351 tablet_copy_source_session.cc:205] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:34.460467 11212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9adf1535c974b2abf910f3c1c537627. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:34.479391 11212 tablet_copy_client.cc:791] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:34.480093 11212 tablet_copy_client.cc:655] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:34.484099 11212 tablet_copy_client.cc:531] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:34.493384 11212 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:34.510295 11212 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:34.511075 11212 tablet_bootstrap.cc:492] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:34.511739 11212 ts_tablet_manager.cc:1367] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.003s
I20240507 08:05:34.514513 11212 raft_consensus.cc:348] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.515153 11212 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:34.515745 11212 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.518770 11212 ts_tablet_manager.cc:1398] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20240507 08:05:34.520834 10351 tablet_copy_service.cc:342] P 88a26dac37fa4243b8c2b3d47f403488: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-e9adf1535c974b2abf910f3c1c537627 received from {username='slave'} at 127.0.0.1:47396
I20240507 08:05:34.521281 10351 tablet_copy_service.cc:433] P 88a26dac37fa4243b8c2b3d47f403488: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-e9adf1535c974b2abf910f3c1c537627 on tablet e9adf1535c974b2abf910f3c1c537627 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:34.685285 10259 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:34.705847 10259 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:34.708425 11188 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.709233 11212 ts_tablet_manager.cc:902] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:34.713604 11212 tablet_copy_client.cc:316] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.131:37047
I20240507 08:05:34.715471 10351 tablet_copy_service.cc:140] P 88a26dac37fa4243b8c2b3d47f403488: Received BeginTabletCopySession request for tablet 8f7d05e73be847e99048092c9570faed from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:47396)
I20240507 08:05:34.716084 10351 tablet_copy_service.cc:161] P 88a26dac37fa4243b8c2b3d47f403488: Beginning new tablet copy session on tablet 8f7d05e73be847e99048092c9570faed from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:47396: session id = 5d6656a616b747d393ff3de5328d2540-8f7d05e73be847e99048092c9570faed
I20240507 08:05:34.729180 10351 tablet_copy_source_session.cc:205] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:34.732470 11212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7d05e73be847e99048092c9570faed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:34.738529 11219 ts_tablet_manager.cc:902] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:34.742002 10190 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:34.744227 11188 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:34.772353 11219 tablet_copy_client.cc:316] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.131:37047
I20240507 08:05:34.774015 10351 tablet_copy_service.cc:140] P 88a26dac37fa4243b8c2b3d47f403488: Received BeginTabletCopySession request for tablet 84da12b367b046b7bcfb08aaa6e1c97b from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:47396)
I20240507 08:05:34.774626 10351 tablet_copy_service.cc:161] P 88a26dac37fa4243b8c2b3d47f403488: Beginning new tablet copy session on tablet 84da12b367b046b7bcfb08aaa6e1c97b from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:47396: session id = 5d6656a616b747d393ff3de5328d2540-84da12b367b046b7bcfb08aaa6e1c97b
I20240507 08:05:34.781955 11212 tablet_copy_client.cc:791] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:34.782755 11212 tablet_copy_client.cc:655] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:34.794806 11212 tablet_copy_client.cc:531] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:34.801749 10351 tablet_copy_source_session.cc:205] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:34.811254 11219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da12b367b046b7bcfb08aaa6e1c97b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:34.820183 11212 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:34.827095 11224 ts_tablet_manager.cc:902] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:34.828315 11219 tablet_copy_client.cc:791] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:34.828891 11219 tablet_copy_client.cc:655] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:34.837363 11224 tablet_copy_client.cc:316] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:05:34.839104 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet e0f47233c3be4fe99e097ade2aaec1e8 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:05:34.839632 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet e0f47233c3be4fe99e097ade2aaec1e8 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-e0f47233c3be4fe99e097ade2aaec1e8
I20240507 08:05:34.841387 11219 tablet_copy_client.cc:531] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:34.847489 10277 tablet_copy_source_session.cc:205] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:05:34.851936 11219 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:34.855903 11224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f47233c3be4fe99e097ade2aaec1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:05:34.857122 11212 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:34.858053 11212 tablet_bootstrap.cc:492] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:34.858758 11212 ts_tablet_manager.cc:1367] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.012s
I20240507 08:05:34.861517 11212 raft_consensus.cc:348] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.862429 11212 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:34.863087 11212 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.872594 11212 ts_tablet_manager.cc:1398] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20240507 08:05:34.874132 10351 tablet_copy_service.cc:342] P 88a26dac37fa4243b8c2b3d47f403488: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-8f7d05e73be847e99048092c9570faed received from {username='slave'} at 127.0.0.1:47396
I20240507 08:05:34.874541 10351 tablet_copy_service.cc:433] P 88a26dac37fa4243b8c2b3d47f403488: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-8f7d05e73be847e99048092c9570faed on tablet 8f7d05e73be847e99048092c9570faed with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:34.888351 11224 tablet_copy_client.cc:791] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:05:34.890647 11224 tablet_copy_client.cc:655] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:05:34.895116 11224 tablet_copy_client.cc:531] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:05:34.901840 11219 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:34.902853 11219 tablet_bootstrap.cc:492] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:34.903584 11219 ts_tablet_manager.cc:1367] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.052s user 0.019s sys 0.008s
I20240507 08:05:34.906642 11219 raft_consensus.cc:348] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.907577 11219 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:34.908298 11219 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.910557 11219 ts_tablet_manager.cc:1398] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240507 08:05:34.912199 10351 tablet_copy_service.cc:342] P 88a26dac37fa4243b8c2b3d47f403488: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-84da12b367b046b7bcfb08aaa6e1c97b received from {username='slave'} at 127.0.0.1:47396
I20240507 08:05:34.912601 10351 tablet_copy_service.cc:433] P 88a26dac37fa4243b8c2b3d47f403488: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-84da12b367b046b7bcfb08aaa6e1c97b on tablet 84da12b367b046b7bcfb08aaa6e1c97b with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:34.912703 11224 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:05:34.917382 10942 raft_consensus.cc:1203] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:34.943166 11224 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:05:34.943964 11224 tablet_bootstrap.cc:492] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:05:34.944556 11224 ts_tablet_manager.cc:1367] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.011s
I20240507 08:05:34.947216 11224 raft_consensus.cc:348] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.947978 11224 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:05:34.948587 11224 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:34.951853 11224 ts_tablet_manager.cc:1398] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20240507 08:05:34.954344 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-e0f47233c3be4fe99e097ade2aaec1e8 received from {username='slave'} at 127.0.0.1:39118
I20240507 08:05:34.954766 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-e0f47233c3be4fe99e097ade2aaec1e8 on tablet e0f47233c3be4fe99e097ade2aaec1e8 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:35.092918 11152 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.093374 11152 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.095769 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.095890 11152 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.096091 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.096403 10259 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:35.097257 10337 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:35.097532 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.098286 11152 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.098637 11152 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.098974 11152 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.108601 11152 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.110945 11152 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.111881 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.112453 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.113211 10337 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.113215 10259 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.119920 10337 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.120913 10156 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.121922 11152 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.123054 10259 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.130071 11152 raft_consensus.cc:686] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:35.131034 11152 consensus_queue.cc:237] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.163074 11180 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.163653 11180 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.165529 11180 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.166850 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.167687 10339 raft_consensus.cc:2379] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.169169 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.169814 10189 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.171010 10232 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:35.173012 11180 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:35.173416 11180 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.173833 11180 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:35.182278 11180 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.184379 11180 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.185000 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.186125 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.193040 10339 raft_consensus.cc:2379] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.199345 10189 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.200573 10232 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:35.201635 11160 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.202121 11160 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.204082 11160 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.204574 11160 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:35.204989 11160 raft_consensus.cc:686] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:35.206581 11160 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.207437 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.208139 10190 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.209173 10232 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.211757 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.212559 10335 raft_consensus.cc:2379] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.216002 11180 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:35.216403 11180 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.216800 11180 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:35.235561 10942 raft_consensus.cc:1203] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:35.238045 10090 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:35.242043 11160 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.251641 11180 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.242563 11160 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.255466 11160 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.255610 11180 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.258388 11180 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.258874 11180 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.260679 11180 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:35.279697 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 68987.1us
I20240507 08:05:35.286863 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.287396 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.287526 10190 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.297326 10189 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.299480 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.300078 10189 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.319607 10232 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.321409 10086 catalog_manager.cc:5484] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:35.323427 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.324153 10335 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.325845 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.326473 10335 raft_consensus.cc:2379] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.327181 11231 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:35.327625 11231 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.327788 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.328557 10335 raft_consensus.cc:2379] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.328013 11231 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:35.332144 11234 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.332633 11234 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.334617 11234 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.339102 10942 raft_consensus.cc:1203] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:35.347438 10942 raft_consensus.cc:1203] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240507 08:05:35.351948 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.352569 10337 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.355852 11231 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.356808 11188 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.362068 11188 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.363076 11231 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.328680 10232 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.364603 11188 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.367481 11234 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:35.368068 11234 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:35.368338 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.356719 11152 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.368932 11234 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.369371 11152 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.373167 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.373867 10337 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:35.373915 10189 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:35.374899 10156 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.375263 10306 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:35.376430 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 47864us
I20240507 08:05:35.377139 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:35.381130 11233 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:35.381522 11233 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.381870 11188 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.381958 11233 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:35.382668 11188 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.383172 11188 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.397773 11229 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.398380 11229 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.398583 11233 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.400708 11233 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.401454 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.402094 10337 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:35.397440 11239 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.402873 11239 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.403215 11239 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.403462 10156 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.404318 11229 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.404942 11229 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.405398 11229 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.405867 11229 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:35.410411 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 55798.1us
W20240507 08:05:35.410893 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 53353.1us
I20240507 08:05:35.397838 11152 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.417462 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.419028 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.420374 10260 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:35.401170 11188 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.421509 10090 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:35.423713 11188 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.425045 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.414405 11229 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.425693 10190 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.427798 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.428351 10260 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.431154 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.432705 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.433310 10189 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.435726 10260 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.436401 11231 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.436780 11231 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.436832 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.437916 11152 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.438336 11229 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.439049 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.439322 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.439922 10337 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.442016 11152 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:35.442935 11152 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.444257 10183 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.445205 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.445557 11241 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.446038 11241 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.446360 10337 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.447976 11241 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.448961 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.449294 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.449924 10260 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:35.450858 10155 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.449568 10337 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:35.451598 11241 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.451987 11241 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.452301 11241 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.455852 10190 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:35.457017 10306 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.458527 11238 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.458981 11238 raft_consensus.cc:686] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:35.456110 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.459764 11238 consensus_queue.cc:237] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.437204 11231 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.463021 11234 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.474224 10090 catalog_manager.cc:5484] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:35.475085 11234 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.471164 11180 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.476263 11180 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.477010 11188 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.477447 11188 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.472008 11160 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.478207 11180 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.483412 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.484067 10184 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:35.485000 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.485109 10306 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.485831 10336 raft_consensus.cc:2379] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.485937 11170 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.486353 11170 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.486443 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.486727 11170 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.487128 10335 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:35.487356 11234 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.485664 11160 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.489041 11241 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.491999 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.492533 10259 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.464823 11239 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.498049 10189 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.499480 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.500268 11160 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.501101 11233 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:35.501626 11233 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:35.502161 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.502751 10258 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.502475 11233 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.505136 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.506335 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.506847 10259 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.507321 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.507953 10335 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.509370 10260 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:35.510617 10155 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.512509 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.496484 11188 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.519942 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.520593 10332 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:35.515022 11152 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.522722 11152 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:35.523830 11152 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.525821 11241 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.526211 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.531311 10335 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.531553 10258 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.532253 10156 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.511682 11239 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.539173 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.539935 10332 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.534341 11241 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.541628 11241 raft_consensus.cc:686] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:35.539921 11170 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.543108 10232 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:35.542367 11241 consensus_queue.cc:237] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.546955 11128 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.547453 11128 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.548066 11248 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:35.548790 10231 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.550426 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.550961 10189 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:35.553500 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.555634 10306 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.558697 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.557937 10090 catalog_manager.cc:5484] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:35.562238 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.559890 11244 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.563689 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.549366 11128 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.572161 10183 raft_consensus.cc:2354] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:35.566773 10186 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:35.570909 11244 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.576103 11244 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.561255 11155 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.578351 11155 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.578689 11155 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.584040 10336 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:35.560815 11160 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:35.585824 11160 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:35.586426 11244 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.586722 11160 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.588428 11244 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.590632 11229 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:35.593267 11170 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.593922 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.594493 10186 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.594841 10332 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:35.595903 11231 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.597815 11231 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.603108 10186 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:35.605779 10306 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.548660 11248 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:35.604909 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56213.5us
I20240507 08:05:35.614288 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.614897 10183 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.614804 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:35.624601 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 67545.6us
I20240507 08:05:35.626945 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:35.629770 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.641140 10183 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:35.667909 10232 leader_election.cc:398] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:35.679103 10232 leader_election.cc:401] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:05:35.678977 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.669173 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:35.673913 10260 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:35.680846 10339 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.679986 11170 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.684793 11229 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.685381 11170 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.685467 11229 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.635362 10184 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.687511 11170 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.679864 10341 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.669698 11180 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:35.691296 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:35.691078 11229 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.691964 10332 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:35.693055 10156 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:35.671182 11247 raft_consensus.cc:1050] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:35.695031 11241 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.695474 11241 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.695890 11241 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.615226 11253 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.702476 10232 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:35.649082 10090 catalog_manager.cc:5484] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:35.707073 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.709982 11231 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.713332 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.713982 10262 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:35.715016 11180 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:35.715116 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.715584 11180 raft_consensus.cc:686] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:35.716419 11180 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.704586 11155 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.719415 11155 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:35.720981 11152 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:35.721853 11247 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:35.724107 11231 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:35.730384 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.734423 10341 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:35.706164 11253 raft_consensus.cc:686] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:35.741752 10088 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20240507 08:05:35.748669 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 143040us
I20240507 08:05:35.751740 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:35.752621 10190 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:35.753657 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:35.754365 10189 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.737877 11152 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.788159 11152 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:35.766834 10189 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:35.814213 10339 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:35.749372 11248 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.742276 11241 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:35.847816 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 57012.7us
I20240507 08:05:35.849709 10183 raft_consensus.cc:2354] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
W20240507 08:05:35.870972 11259 log.cc:924] Time spent T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240507 08:05:35.882369 11152 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.886934 11241 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:35.911684 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:35.913286 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:35.914126 10262 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.912572 10260 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:35.866027 11255 raft_consensus.cc:1050] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:35.947247 10155 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:35.922435 11257 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20240507 08:05:35.954633 10232 leader_election.cc:398] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:35.960881 10232 leader_election.cc:401] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:05:35.963505 11152 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:35.964289 11152 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:35.964735 11152 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:35.978299 10183 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:35.997542 11267 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:35.998129 11267 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.000111 11267 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:36.003347 11248 raft_consensus.cc:1050] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:36.008810 10186 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:36.009099 11248 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:36.014008 10183 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:35.860152 11253 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.031617 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:36.032498 10339 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:36.019724 11244 consensus_queue.cc:1046] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240507 08:05:36.041251 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:36.043658 11244 raft_consensus.cc:3032] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:36.044091 11244 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:36.044898 11244 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.047863 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:36.043592 10336 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:36.046540 10184 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:36.049396 10262 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:36.046571 11244 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:36.052482 11265 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.056s user 0.000s sys 0.001s
I20240507 08:05:36.059079 10339 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:36.060307 10156 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:36.056598 11180 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:36.067587 11233 consensus_queue.cc:1046] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.067730 11231 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.068486 11249 raft_consensus.cc:3032] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:36.068831 11249 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:36.073024 11266 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:36.073462 11266 raft_consensus.cc:686] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:36.074244 11266 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.074709 10232 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:36.075943 11250 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:36.076565 11250 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:36.077414 11250 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.069552 11249 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.090566 11249 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:36.046115 11255 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:36.103240 11152 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.105752 10183 raft_consensus.cc:2379] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:36.111943 10306 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:36.113150 11263 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:36.113520 11263 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:36.113870 11263 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:36.116817 10306 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:36.118029 11269 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
W20240507 08:05:36.120263 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 59838.9us
I20240507 08:05:36.123142 10339 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.120057 11269 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:36.127338 10262 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:05:36.128909 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 397510us
I20240507 08:05:36.134505 11269 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.135269 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:36.135844 10339 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:36.177242 10258 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:05:36.179715 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 43841.6us
I20240507 08:05:36.183740 10339 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:36.175032 11152 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:36.186854 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:36.189119 10262 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:36.190367 10261 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.193931 10258 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.197161 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:36.198879 11239 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:36.198763 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:36.199473 11239 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:36.199543 10258 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:36.200445 11239 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.207612 11273 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.210117 10262 raft_consensus.cc:1261] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.214540 10339 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.126503 11270 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20240507 08:05:36.221076 10190 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.226565 10190 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.240146 10258 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:36.245442 10188 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.245674 11281 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.246271 11247 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.253882 11247 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:36.235320 11241 consensus_queue.cc:1035] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20240507 08:05:36.244056 11229 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240507 08:05:36.278681 11263 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.254490 11250 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20240507 08:05:36.283336 11263 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:36.284782 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:36.298373 10186 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:36.300846 11231 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.302729 10940 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.401121 11128 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:36.423380 10332 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:36.426487 10942 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.427858 10186 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.436587 10341 raft_consensus.cc:1261] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.439067 10183 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.437635 10941 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:36.499541 10341 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.501416 11241 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:36.471001 11279 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240507 08:05:36.471108 11249 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240507 08:05:36.446365 11282 consensus_queue.cc:1035] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240507 08:05:36.542928 11233 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:36.531383 11258 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240507 08:05:36.558522 11268 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:36.559185 11268 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.567292 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:36.598186 10259 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.602274 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:36.615027 11247 raft_consensus.cc:1050] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:36.587284 11268 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:36.631186 10337 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:36.651125 10335 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:36.609395 11152 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20240507 08:05:36.691231 10335 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:36.731350 10189 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.706130 11248 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:36.839717 10339 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240507 08:05:36.847800 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 77011.9us
W20240507 08:05:36.854820 11304 log.cc:924] Time spent T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20240507 08:05:36.856585 10338 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:05:36.854892 11295 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20240507 08:05:36.875871 11249 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:36.876657 11249 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:36.879143 11249 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:36.780786 11180 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20240507 08:05:36.881970 11180 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:36.873448 10337 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.889653 10188 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.916625 10190 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.769812 11248 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.882687 11180 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.950245 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:36.921535 11267 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20240507 08:05:36.934775 11155 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240507 08:05:36.966516 11248 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:36.966353 11238 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240507 08:05:36.968642 11298 log.cc:924] Time spent T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.109s user 0.001s sys 0.000s
I20240507 08:05:36.961553 11128 raft_consensus.cc:2741] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240507 08:05:36.979156 10337 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:36.985726 11250 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:36.986354 11250 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:36.988504 11250 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:36.988655 11303 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.265s user 0.003s sys 0.000s
I20240507 08:05:36.990561 11180 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:36.993855 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 70341.7us
I20240507 08:05:36.996043 11269 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:37.013152 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:36.998144 11231 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.038089 11231 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:37.038430 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:37.035276 11180 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240507 08:05:37.039602 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:37.039949 11231 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:37.040741 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:37.050434 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:37.060851 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:37.063356 10260 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:37.066630 11308 raft_consensus.cc:3032] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:37.092139 11308 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:37.106719 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:37.109736 10260 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240507 08:05:37.110270 10260 pending_rounds.cc:76] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 1
I20240507 08:05:37.110623 10260 pending_rounds.cc:97] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240507 08:05:37.111071 10260 raft_consensus.cc:2866] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:37.115702 10260 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20240507 08:05:37.116253 10260 pending_rounds.cc:76] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 1
I20240507 08:05:37.116580 10260 pending_rounds.cc:97] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20240507 08:05:37.116940 10260 raft_consensus.cc:2866] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:37.124568 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:37.125833 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:37.201144 11301 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.518s user 0.003s sys 0.000s
W20240507 08:05:37.205710 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 190794us
I20240507 08:05:37.225710 11231 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:37.228247 11231 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240507 08:05:37.221815 11314 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.247s user 0.001s sys 0.000s
I20240507 08:05:37.202167 11308 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:37.213860 11248 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240507 08:05:37.224339 11180 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20240507 08:05:37.309815 11319 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.310693 11319 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:37.343421 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:37.387812 10090 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:37.392663 10090 catalog_manager.cc:5484] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:05:37.398216 10090 catalog_manager.cc:5484] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:37.400110 10090 catalog_manager.cc:5484] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20240507 08:05:37.404685 11322 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.094s user 0.001s sys 0.001s
I20240507 08:05:37.427840 11310 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240507 08:05:37.431152 11238 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:37.428885 11247 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:37.452934 11308 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:37.475827 11315 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:37.374737 11319 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:37.503585 10942 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:05:37.517079 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 391530us
I20240507 08:05:37.615783 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:37.625419 10258 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:05:37.633663 11301 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.060s user 0.002s sys 0.005s
I20240507 08:05:37.688582 11312 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.826105 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:37.809842 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 46576.8us
I20240507 08:05:37.830757 10233 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:37.895238 11331 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.824580 11327 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.903075 11331 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:37.903528 11327 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:37.725675 11255 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:37.895251 11233 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.905979 11255 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:37.906520 11233 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:37.908016 11255 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:37.909121 11233 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:37.821681 11271 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:37.910102 11248 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:05:37.910928 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 80319.1us
I20240507 08:05:37.895534 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:37.764304 11320 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20240507 08:05:37.753921 11221 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:37.755527 11250 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:37.727661 11180 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.015862 11319 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.038486 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 708) took 1005 ms. Trace:
I20240507 08:05:38.137123 11331 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.017731 11250 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:38.181924 10942 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:38.139647 10192 rpcz_store.cc:276] 0507 08:05:37.033165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.033283 (+ 118us) service_pool.cc:225] Handling call
0507 08:05:37.033872 (+ 589us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:37.034184 (+ 312us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:37.034209 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:37.035080 (+ 871us) log.cc:842] Serialized 564 byte log entry
0507 08:05:37.260262 (+225182us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:37.260332 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:37.260444 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:38.038197 (+777753us) raft_consensus.cc:1671] finished
0507 08:05:38.038220 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:38.038409 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":224729,"threads_started":1,"wal-append.queue_time_us":538355}
I20240507 08:05:38.133332 11180 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.159619 11280 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.048072 11221 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:38.175668 11266 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.150501 11319 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.224665 11331 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:38.225338 11331 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.199894 11280 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.246920 11280 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.252168 11221 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.217897 11266 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.176982 11327 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.257570 11266 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.459349 11270 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240507 08:05:38.463366 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 636198us
I20240507 08:05:38.465739 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1022) took 1408 ms. Trace:
I20240507 08:05:38.466274 10186 rpcz_store.cc:276] 0507 08:05:37.057192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.057495 (+ 303us) service_pool.cc:225] Handling call
0507 08:05:37.057909 (+ 414us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:37.533178 (+475269us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:37.533204 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:37.534532 (+ 1328us) log.cc:842] Serialized 26 byte log entry
0507 08:05:37.687903 (+153371us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:37.687943 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:37.688012 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:38.465443 (+777431us) raft_consensus.cc:1671] finished
0507 08:05:38.465470 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:38.465647 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8949,"raft.queue_time_us":69898,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":5265,"thread_start_us":572361,"threads_started":2,"wal-append.queue_time_us":603726}
W20240507 08:05:38.493458 11325 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.011s sys 0.019s
I20240507 08:05:37.841432 11312 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.508648 11280 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.508854 11266 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:38.551765 11325 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s user 0.011s sys 0.019s
I20240507 08:05:38.551002 11255 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.549813 11319 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.562480 11336 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:38.576527 11319 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:38.602571 11359 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.603278 11359 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.605696 11359 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:38.606302 11359 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.606832 11359 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.609068 11359 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:38.566028 11266 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.615439 11266 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.581626 11336 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.618073 11336 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.619941 11180 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.621865 11356 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:38.622697 11344 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240507 08:05:38.622452 11356 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.624532 11356 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.569555 11280 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.574873 11255 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:38.644918 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:38.591426 10337 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.038s sys 0.052s
W20240507 08:05:38.649299 10337 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.039s sys 0.052s
I20240507 08:05:38.652961 11355 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.631500 11352 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.662349 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 720) took 1005 ms. Trace:
I20240507 08:05:38.651504 11180 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.632259 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.651065 11336 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.683298 11266 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.683898 11336 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.684049 11266 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.686288 11336 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.688373 11266 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.690696 11255 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:38.680331 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:38.657938 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 760120us
I20240507 08:05:38.663676 10339 rpcz_store.cc:276] 0507 08:05:37.657492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.657664 (+ 172us) service_pool.cc:225] Handling call
0507 08:05:38.661472 (+1003808us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:38.661724 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:38.661936 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:38.661972 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:38.662037 (+ 65us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:38.662081 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:38.662152 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:38.662283 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33881984}
I20240507 08:05:38.626648 11319 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.683187 11345 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.682982 11180 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.694008 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.659340 11327 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.712793 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.656023 11253 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.661420 11325 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.714977 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.712630 10190 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:38.718652 10192 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:38.723922 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.711027 11351 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:38.728335 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.719348 11309 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.720458 11253 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:38.732690 11350 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240507 08:05:38.677834 11355 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.723264 11280 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.733831 11253 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:38.734498 11355 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.724858 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 713) took 1134 ms. Trace:
I20240507 08:05:38.735373 11180 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.720546 11325 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.735867 11180 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.736078 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:38.736366 11180 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.737272 11325 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.707839 11345 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:38.735862 11319 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.680712 11352 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.738596 11309 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.749467 11309 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:38.749771 11337 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.529s user 0.005s sys 0.005s
I20240507 08:05:38.751133 11180 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.752350 11361 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:38.740826 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.721184 11327 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.729038 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.729964 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:38.746646 11319 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.730142 11351 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.735080 10332 rpcz_store.cc:276] 0507 08:05:37.587423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.615623 (+ 28200us) service_pool.cc:225] Handling call
0507 08:05:38.631110 (+1015487us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec572e 000000000033cd89 00007fcf95cf46c0 00007fcf95dba3c4 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:38.720682 (+ 89572us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7945984}
I20240507 08:05:38.755071 10183 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:38.760735 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.761932 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:38.762462 10339 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:38.763382 10156 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:38.757417 11336 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.759793 11319 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.764997 11336 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:38.766814 11319 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.059638 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:38.849431 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 90814.2us
W20240507 08:05:38.816955 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 53645.7us
I20240507 08:05:39.063556 11238 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.064181 11238 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.066330 11238 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:39.067608 11238 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.068159 11238 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.071122 10188 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240507 08:05:39.003427 11372 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20240507 08:05:39.091130 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:38.819391 10231 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.555s (SENT) [suppressed 3 similar messages]
I20240507 08:05:39.091337 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.092355 10183 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:38.766891 11336 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.757788 11180 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.845118 11312 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.839857 11325 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.790302 11351 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.790932 11345 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.789482 11361 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.833490 11355 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.819543 11270 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.773898 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:38.846212 11327 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:38.928864 11353 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:38.993075 11310 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:38.841266 11352 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.111553 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:39.252115 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 256419us
I20240507 08:05:39.229174 11369 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.252244 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:39.251827 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 189240us
I20240507 08:05:39.255831 10232 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:39.253223 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.229429 11269 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.274121 11357 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.274562 11269 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.274668 11357 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.276585 11357 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:39.229732 11128 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.277642 11382 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:39.278291 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 459121us
I20240507 08:05:39.283859 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:38.842427 11368 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:39.325970 10183 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:39.326217 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.328251 10188 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:39.329804 11352 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.330149 11365 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.330464 11352 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.330873 11365 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.329622 11351 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.162958 11238 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:39.333370 11365 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:39.282743 11327 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.157816 11280 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.156637 11319 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.148598 11373 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.334122 11315 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:39.333494 11352 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:39.342339 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 228076us
I20240507 08:05:39.282510 11353 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.341193 11128 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.334391 11368 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.374280 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.403481 10263 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:39.116367 11366 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:39.354563 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 100749us
I20240507 08:05:39.327337 11312 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.336432 11345 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.387528 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.341995 11336 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:39.276597 11269 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:39.112586 11180 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.115974 11359 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.236s
I20240507 08:05:39.250003 11358 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.271917 11325 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.282691 11355 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.346635 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.253253 11369 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.352632 11309 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.282944 11310 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.417670 11378 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.253803 11270 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.342113 11351 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.421828 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.373811 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 718) took 1708 ms. Trace:
I20240507 08:05:39.420866 11370 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.368638 11382 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.419881 11266 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.430786 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.284662 11361 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.414587 11352 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.405275 11368 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:39.398627 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 144934us
I20240507 08:05:39.386215 11280 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.392768 11319 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.421247 11336 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:39.442656 11377 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.429845 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.472965 10263 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:39.402107 11315 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.393503 11375 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.430049 11312 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.471243 10258 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:39.429430 11366 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.428504 11345 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.435921 11309 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.392932 11373 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.401003 11327 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:39.586233 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:39.587141 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 201909us
I20240507 08:05:39.585951 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.900127 10262 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:39.603150 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.585619 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:39.899227 10260 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:39.611833 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 721) took 2841 ms. Trace:
I20240507 08:05:39.897842 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 728) took 2000 ms. Trace:
I20240507 08:05:39.880616 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.896960 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.898553 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.610152 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 726) took 1816 ms. Trace:
I20240507 08:05:39.955128 10188 raft_consensus.cc:2396] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:39.973815 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 744) took 1036 ms. Trace:
I20240507 08:05:39.941869 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:39.902386 10259 rpcz_store.cc:276] 0507 08:05:36.769998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:36.770673 (+ 675us) service_pool.cc:225] Handling call
0507 08:05:36.771052 (+ 379us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:36.858507 (+ 87455us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:36.858532 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:36.872204 (+ 13672us) log.cc:842] Serialized 26 byte log entry
0507 08:05:37.345352 (+473148us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b1800043d80. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:05:37.586381 (+241029us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:37.586412 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:37.586462 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:39.610937 (+2024475us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b24002a90c8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dd58024 00007fcf8dd5613b 00007fcf8dde4149 00007fcf8dde3dd5 00007fcf8dde3330 00007fcf90ad63a8 00007fcf90ad6b08 00007fcf90aae618 00007fcf90aa9a1a 00007fcf90aa8e66
0507 08:05:39.611600 (+ 663us) raft_consensus.cc:1671] finished
0507 08:05:39.611630 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.611756 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":87036,"raft.queue_time_us":96881,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":132984832,"thread_start_us":240876,"threads_started":1,"wal-append.queue_time_us":749333}
I20240507 08:05:39.560968 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.481678 11336 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:39.958508 10189 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:40.052248 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.479476 11270 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:40.053961 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 112279us
I20240507 08:05:39.904172 10341 rpcz_store.cc:276] 0507 08:05:37.897020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:38.629705 (+732685us) service_pool.cc:225] Handling call
0507 08:05:38.629989 (+ 284us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:38.655422 (+ 25433us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:38.655465 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:38.655516 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:38.655626 (+ 110us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.838157 (+1182531us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b540013f454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:39.838228 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.897772 (+ 59544us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58062208}
W20240507 08:05:39.577136 11389 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20240507 08:05:40.052866 10262 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:39.433916 11358 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.056939 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:40.057641 10261 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.057s sys 0.114s
W20240507 08:05:40.066426 10261 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s user 0.057s sys 0.114s
I20240507 08:05:39.616814 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.052059 11359 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.527833 11375 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.070660 11361 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:39.615928 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 717) took 1977 ms. Trace:
I20240507 08:05:40.071552 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.607131 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:39.619025 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.493304 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.586647 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 722) took 1865 ms. Trace:
I20240507 08:05:39.621201 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 743) took 1712 ms. Trace:
I20240507 08:05:40.092285 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:39.609268 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 190803us
I20240507 08:05:39.935302 10183 raft_consensus.cc:2396] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240507 08:05:39.606542 11387 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240507 08:05:40.058239 11325 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.063819 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.064625 11362 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:39.930858 10338 rpcz_store.cc:276] 0507 08:05:37.829477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.913174 (+ 83697us) service_pool.cc:225] Handling call
0507 08:05:37.913487 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:38.509960 (+596473us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:38.509992 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:38.510684 (+ 692us) log.cc:842] Serialized 26 byte log entry
0507 08:05:39.110078 (+599394us) spinlock_profiling.cc:243] Waited 4.04 ms on lock 0x7b18001fd620. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:05:39.373754 (+263676us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:39.481976 (+108222us) log.cc:842] Serialized 14 byte log entry
0507 08:05:39.609787 (+127811us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.609866 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:39.609907 (+ 41us) raft_consensus.cc:1671] finished
0507 08:05:39.609985 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.610101 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":723674,"prepare.queue_time_us":97299,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":770005,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":2688,"spinlock_wait_cycles":9628288,"thread_start_us":322035,"threads_started":2,"wal-append.queue_time_us":891330}
I20240507 08:05:39.518019 11369 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.837087 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:40.066247 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:39.476393 11352 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.071650 11384 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.616494 11315 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:39.976697 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.071223 10335 rpcz_store.cc:276] 0507 08:05:37.638195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.638340 (+ 145us) service_pool.cc:225] Handling call
0507 08:05:39.615088 (+1976748us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:39.615339 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:39.615556 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:39.615582 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:39.615629 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:39.615672 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.615740 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.615864 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7989632}
I20240507 08:05:39.895273 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 729) took 1023 ms. Trace:
I20240507 08:05:39.879835 11379 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.896183 11331 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.560863 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.976120 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.079691 11366 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.974578 10188 rpcz_store.cc:276] 0507 08:05:38.936898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.880279 (+943381us) service_pool.cc:225] Handling call
0507 08:05:39.973735 (+ 93456us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.134956 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:39.609019 11263 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:39.813750 11128 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.096143 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 26) took 1295 ms. Trace:
I20240507 08:05:40.150189 10339 rpcz_store.cc:276] 0507 08:05:38.791465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.616518 (+825053us) service_pool.cc:225] Handling call
0507 08:05:40.086433 (+469915us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:39.561161 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:39.582176 11250 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.622201 11271 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:39.560467 11383 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.110503 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 748) took 1171 ms. Trace:
I20240507 08:05:39.891692 11368 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.157491 10183 rpcz_store.cc:276] 0507 08:05:38.940399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.602801 (+662402us) service_pool.cc:225] Handling call
0507 08:05:40.110442 (+507641us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.161051 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.161686 10339 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:40.066767 11270 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.162170 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 739) took 1230 ms. Trace:
I20240507 08:05:39.473728 11266 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.092566 10337 rpcz_store.cc:276] 0507 08:05:37.909286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.089260 (+1179974us) service_pool.cc:225] Handling call
0507 08:05:39.089569 (+ 309us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:39.089789 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:39.152211 (+ 62422us) log.cc:842] Serialized 14 byte log entry
0507 08:05:39.620668 (+468457us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b1800198940. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf90ab675c 00007fcf8dedf54c 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8def9cc5 00007fcf90e1f620 00007fcf90e317c6 00007fcf90e42327
0507 08:05:39.620786 (+ 118us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:39.620846 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:39.620900 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.620996 (+ 96us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.621137 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":333524,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":72810,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":9844,"spinlock_wait_cycles":3682560,"thread_start_us":224539,"threads_started":3,"wal-append.queue_time_us":277644}
I20240507 08:05:40.127100 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:40.165724 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 507358us
I20240507 08:05:40.072614 10185 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:39.529063 10908 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:40.135443 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:40.120755 11351 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.084615 11359 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.138860 11400 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.089007 11353 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.138319 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 758) took 1009 ms. Trace:
I20240507 08:05:40.153267 11375 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.080526 11258 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20240507 08:05:40.127139 11325 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.161207 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.187806 10335 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:40.087637 11365 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20240507 08:05:40.162510 10183 rpcz_store.cc:276] 0507 08:05:38.931343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.161497 (+1230154us) service_pool.cc:225] Handling call
0507 08:05:40.161770 (+ 273us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.161919 (+ 149us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.161943 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.161978 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.162002 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.162044 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.162128 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.134212 11369 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:40.169319 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 745) took 1507 ms. Trace:
I20240507 08:05:40.171485 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 25) took 1510 ms. Trace:
I20240507 08:05:40.198709 10339 rpcz_store.cc:276] 0507 08:05:38.660701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.168255 (+1507554us) service_pool.cc:225] Handling call
0507 08:05:40.171049 (+ 2794us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.171203 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.171226 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.171260 (+ 34us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:05:40.171287 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.171334 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.171431 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:39.527469 11377 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.138336 11395 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:40.176833 10186 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:40.151479 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 744) took 2236 ms. Trace:
I20240507 08:05:40.152122 11356 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.178097 10338 rpcz_store.cc:276] 0507 08:05:39.128991 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.100974 (+971983us) service_pool.cc:225] Handling call
0507 08:05:40.138255 (+ 37281us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.181283 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.181512 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.136384 11362 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.228791 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:40.182651 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 746) took 1245 ms. Trace:
I20240507 08:05:40.149839 10266 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:40.231315 10189 rpcz_store.cc:276] 0507 08:05:38.937484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.181974 (+1244490us) service_pool.cc:225] Handling call
0507 08:05:40.182226 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.182377 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.182400 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.182436 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.182461 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.182507 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.182606 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.183670 11375 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.092103 10340 rpcz_store.cc:276] 0507 08:05:37.721460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.729612 (+ 8152us) service_pool.cc:225] Handling call
0507 08:05:39.585484 (+1855872us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:39.585795 (+ 311us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:39.586036 (+ 241us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:39.586086 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:39.586160 (+ 74us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:39.586212 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.586292 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.586448 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2902272}
I20240507 08:05:40.159994 11128 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:40.150339 10336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 714) took 2544 ms (client timeout 1893 ms). Trace:
I20240507 08:05:40.189687 11310 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.168660 11280 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.197939 10337 rpcz_store.cc:276] 0507 08:05:38.661895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.168554 (+1506659us) service_pool.cc:225] Handling call
0507 08:05:40.168843 (+ 289us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.169002 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.169025 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.169063 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.169091 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.169141 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.169259 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:39.477973 11378 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.492715 11370 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.157039 11271 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.155287 11263 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:40.263063 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 135184us
I20240507 08:05:40.173993 11250 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.184478 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.215451 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 746) took 1525 ms. Trace:
I20240507 08:05:40.167987 11270 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.174186 11188 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:40.196308 11319 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:40.187281 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 695759us
I20240507 08:05:40.227118 10267 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:40.205276 11358 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.234510 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 750) took 1543 ms. Trace:
I20240507 08:05:40.247041 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1062) took 1096 ms. Trace:
I20240507 08:05:40.218402 11357 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.248301 10341 raft_consensus.cc:2379] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:40.186470 11400 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.284919 10189 rpcz_store.cc:276] 0507 08:05:39.150238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.246168 (+1095930us) service_pool.cc:225] Handling call
0507 08:05:40.246543 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.246696 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.246728 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.246776 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.246806 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.246865 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.246979 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.132117 10191 rpcz_store.cc:276] 0507 08:05:38.872264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:38.878869 (+ 6605us) service_pool.cc:225] Handling call
0507 08:05:38.964830 (+ 85961us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:39.476303 (+511473us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:39.481975 (+ 5672us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:39.485024 (+ 3049us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:39.493167 (+ 8143us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.894939 (+401772us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b5400261954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:39.895004 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.895194 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1102767360}
I20240507 08:05:40.252208 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 757) took 1508 ms. Trace:
I20240507 08:05:40.287973 10338 rpcz_store.cc:276] 0507 08:05:38.743736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.227015 (+1483279us) service_pool.cc:225] Handling call
0507 08:05:40.251614 (+ 24599us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.251824 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.251860 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.251913 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.251955 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.252021 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.252144 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:40.250519 10336 rpcz_store.cc:269] 0507 08:05:37.587727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.636916 (+ 49189us) service_pool.cc:225] Handling call
0507 08:05:39.606968 (+1970052us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:40.132320 (+525352us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38969728}
I20240507 08:05:40.289214 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.217506 11373 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.254849 10156 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:40.198582 11325 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.289852 11368 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:39.657905 10941 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:40.201478 11369 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:39.560674 11392 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:40.242198 11390 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:40.264784 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:40.300906 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 46183.8us
I20240507 08:05:40.301000 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.243686 11327 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.154911 11384 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.246762 11388 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.273211 10232 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:40.245698 11401 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:40.274405 10231 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.932s user 0.006s sys 0.021s
I20240507 08:05:40.273744 10335 rpcz_store.cc:276] 0507 08:05:38.689898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.188540 (+1498642us) service_pool.cc:225] Handling call
0507 08:05:40.214813 (+ 26273us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.215036 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.215074 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.215127 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.215170 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.215241 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.215383 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:40.279410 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 150504us
I20240507 08:05:40.254215 11394 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.283985 10339 rpcz_store.cc:276] 0507 08:05:38.691008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.199894 (+1508886us) service_pool.cc:225] Handling call
0507 08:05:40.233879 (+ 33985us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.234092 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.234133 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.234186 (+ 53us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.234231 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.234299 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.234448 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.285400 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.309957 10232 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:40.291071 10264 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240507 08:05:40.291455 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.289775 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.282840 10942 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:40.312775 10187 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:40.276875 11370 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:40.313925 10338 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:40.298938 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.160493 11383 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.250541 11356 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.306425 11353 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.305266 10156 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:40.326174 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.304272 11352 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.277295 11351 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.154158 11379 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.289956 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.245951 11395 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:40.309324 10231 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.967s user 0.007s sys 0.021s
I20240507 08:05:40.258488 11312 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.298197 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:40.314947 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:40.248914 11375 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.278615 11250 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.222859 10332 rpcz_store.cc:276] 0507 08:05:37.914877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.617792 (+1702915us) service_pool.cc:225] Handling call
0507 08:05:40.063294 (+445502us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.080845 (+ 17551us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.087560 (+ 6715us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.099418 (+ 11858us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.110680 (+ 11262us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.128957 (+ 18277us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.139655 (+ 10698us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:40.269642 11310 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.302589 11345 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.322806 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:39.436115 10334 rpcz_store.cc:276] 0507 08:05:37.638531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.651672 (+ 13141us) service_pool.cc:225] Handling call
0507 08:05:38.621154 (+969482us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:38.621436 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:38.622007 (+ 571us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:38.622052 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:38.622742 (+ 690us) log.cc:842] Serialized 26 byte log entry
0507 08:05:38.728500 (+105758us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:39.089309 (+360809us) log.cc:842] Serialized 14 byte log entry
0507 08:05:39.230231 (+140922us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:39.254739 (+ 24508us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:39.272644 (+ 17905us) raft_consensus.cc:1671] finished
0507 08:05:39.339726 (+ 67082us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:39.347084 (+ 7358us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":107493,"prepare.queue_time_us":396250,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":71307,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":54435,"spinlock_wait_cycles":21895680,"thread_start_us":424916,"threads_started":2,"wal-append.queue_time_us":144949}
I20240507 08:05:40.327435 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.327906 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.328876 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.288440 11188 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.253847 11128 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.296801 11357 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.306470 11407 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.320672 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:40.330163 11370 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.328768 11353 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.328042 11394 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.313668 11405 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.329830 11258 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.354576 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.355931 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:40.333484 11266 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.357801 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.359817 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.341806 11352 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:40.321025 11388 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:40.336687 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:40.374941 11402 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20240507 08:05:40.341281 11325 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.316447 11401 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.385835 11395 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.387053 11378 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.331785 11351 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:40.389482 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50337.1us
I20240507 08:05:40.368825 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:40.356148 11345 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.362807 11370 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.398967 11238 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:40.399750 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:40.357658 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 61061.3us
I20240507 08:05:40.403283 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.360777 10184 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:40.371084 10943 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:40.383154 11405 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:40.451717 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45375.5us
I20240507 08:05:40.370220 10191 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:40.451964 11270 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.388384 11403 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.395287 11352 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:40.456588 11408 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:40.457037 11398 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.144s user 0.000s sys 0.001s
W20240507 08:05:40.457445 11399 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20240507 08:05:40.392097 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.362377 11358 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240507 08:05:40.390455 11377 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.370019 11353 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:40.469234 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 100572us
I20240507 08:05:40.487432 11352 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:40.394503 11266 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.352138 11310 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.472119 10339 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:40.487259 11270 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.469493 10086 catalog_manager.cc:5484] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:40.384478 11359 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:40.455281 11400 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.455497 11263 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.468209 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.471908 11370 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:40.456151 11395 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.490181 10232 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:40.398383 11258 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:40.482105 11369 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:40.480240 11405 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:40.398761 11368 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.471783 11345 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.405920 11325 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.483630 11362 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.492211 11403 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.363297 11407 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.698954 10306 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:40.525205 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.672160 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 735) took 1268 ms. Trace:
I20240507 08:05:40.723508 10156 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:40.508730 11356 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:40.717720 10261 rpcz_store.cc:276] 0507 08:05:39.403908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.404267 (+ 359us) service_pool.cc:225] Handling call
0507 08:05:39.404595 (+ 328us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:39.416351 (+ 11756us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:39.416411 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:39.426047 (+ 9636us) log.cc:842] Serialized 26 byte log entry
0507 08:05:40.066846 (+640799us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.273941 (+207095us) log.cc:842] Serialized 14 byte log entry
0507 08:05:40.277556 (+ 3615us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.277644 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:40.671858 (+394214us) raft_consensus.cc:1671] finished
0507 08:05:40.671886 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.672061 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9627,"prepare.queue_time_us":250544,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":58396,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":23346,"thread_start_us":778365,"threads_started":2,"wal-append.queue_time_us":747297}
W20240507 08:05:40.743244 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 244581us
I20240507 08:05:40.751793 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.756433 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.547434 11357 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:40.772495 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 50172.5us
I20240507 08:05:40.783878 11420 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.581565 11375 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.794394 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.660178 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.660522 11383 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.735186 11247 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.547755 11310 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:40.569891 11412 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:40.709610 11384 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.795967 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:40.721524 11403 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:40.746364 11250 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.736225 10336 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:40.545819 11266 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.753720 11393 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:40.721992 11270 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.719677 11325 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.508479 11408 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.791391 11379 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.884814 10339 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:40.812629 11420 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.820513 11247 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:40.886509 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 721) took 3166 ms. Trace:
I20240507 08:05:40.784405 11368 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.739320 11395 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:40.696555 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:40.901324 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 82483.1us
I20240507 08:05:40.851459 11366 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:40.817778 11180 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:40.906512 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 234433us
I20240507 08:05:40.907384 10231 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:40.908547 11404 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:40.899350 10333 rpcz_store.cc:276] 0507 08:05:37.720774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.722967 (+ 2193us) service_pool.cc:225] Handling call
0507 08:05:39.885745 (+2162778us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:40.885397 (+999652us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec56dc 00007fcf8dec529b 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:40.885694 (+ 297us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.885924 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.885964 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.886131 (+ 167us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.886182 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:40.886265 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:40.886407 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":222098944}
I20240507 08:05:40.913332 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.863044 11188 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:40.912995 10263 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.013s sys 0.015s
I20240507 08:05:40.926231 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:40.899214 11247 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:40.904328 10332 raft_consensus.cc:2396] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240507 08:05:40.927896 10263 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s user 0.013s sys 0.015s
I20240507 08:05:40.896914 11420 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:40.902619 11351 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:40.928391 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:40.904678 11384 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:40.939833 10189 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.030s sys 0.019s
I20240507 08:05:40.910907 11366 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.911296 11400 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.915663 11128 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:40.937897 11421 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240507 08:05:40.942535 11366 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:40.941872 10189 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.589s user 0.030s sys 0.019s
I20240507 08:05:40.602669 10265 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:40.929807 10263 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:40.944196 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:40.946874 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 456961us
I20240507 08:05:40.927294 11266 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.926672 11383 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.928828 11388 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.910647 11401 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.937956 11345 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:40.941948 11325 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.944877 11270 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:40.959465 10231 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:40.963346 11401 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:40.967309 11405 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.914084 11407 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:40.978224 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 192436us
W20240507 08:05:40.983072 11336 consensus_meta.cc:287] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.918s user 0.014s sys 0.003s
W20240507 08:05:41.040318 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 80954.3us
I20240507 08:05:41.044389 10156 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:40.947468 11400 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.053364 11428 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.056258 11310 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:41.056665 11310 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:41.057019 11310 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:40.947255 11384 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:40.979497 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.062733 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1063) took 1810 ms. Trace:
I20240507 08:05:41.063177 10189 rpcz_store.cc:276] 0507 08:05:39.252788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.334321 (+1081533us) service_pool.cc:225] Handling call
0507 08:05:40.334579 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.334774 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:40.947578 (+612804us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.062407 (+114829us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.062459 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:41.062499 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.062574 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.062680 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":709131,"threads_started":2}
I20240507 08:05:41.065152 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:40.958904 11383 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:40.974611 11258 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:41.076030 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 86194us
I20240507 08:05:41.062301 11394 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.043905 11336 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.057647 11401 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.377106 10231 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.209295 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 736) took 2279 ms. Trace:
I20240507 08:05:41.381050 10231 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.379496 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1025) took 1085 ms. Trace:
I20240507 08:05:41.381989 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.380573 10192 rpcz_store.cc:276] 0507 08:05:38.930440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.125967 (+1195527us) service_pool.cc:225] Handling call
0507 08:05:40.126381 (+ 414us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.475992 (+349611us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.476037 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.476086 (+ 49us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.476118 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.208950 (+732832us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b540033f4d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.209014 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.209215 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":804004096}
W20240507 08:05:41.227905 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 103096us
I20240507 08:05:41.381978 10263 rpcz_store.cc:276] 0507 08:05:40.294475 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.385034 (+ 90559us) service_pool.cc:225] Handling call
0507 08:05:40.386254 (+ 1220us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:40.386691 (+ 437us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.386729 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:40.402467 (+ 15738us) log.cc:842] Serialized 564 byte log entry
0507 08:05:40.929688 (+527221us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:41.379008 (+449320us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.379224 (+ 216us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.379289 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:41.379327 (+ 38us) raft_consensus.cc:1671] finished
0507 08:05:41.379338 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.379447 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":526628,"threads_started":1,"wal-append.queue_time_us":505154}
I20240507 08:05:41.387367 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.388342 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.388909 10263 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:41.389685 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.390614 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.054409 11408 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:41.210148 11392 consensus_meta.cc:287] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.834s user 0.011s sys 0.006s
I20240507 08:05:41.119096 11407 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:41.341452 11353 consensus_meta.cc:287] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.774s user 0.005s sys 0.006s
I20240507 08:05:41.393110 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 736) took 1075 ms. Trace:
I20240507 08:05:41.404516 10338 rpcz_store.cc:276] 0507 08:05:40.317609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.041814 (+724205us) service_pool.cc:225] Handling call
0507 08:05:41.393042 (+351228us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.119084 11270 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.118507 11383 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.396955 10306 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.403188 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.334575 11384 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.386576 11404 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.121341 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:41.375720 11429 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20240507 08:05:41.413007 11270 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.415051 11270 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.408263 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.378494 11368 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.405715 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.397293 11373 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:41.423223 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 740) took 1104 ms. Trace:
I20240507 08:05:41.428576 10338 rpcz_store.cc:276] 0507 08:05:40.318747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.405469 (+1086722us) service_pool.cc:225] Handling call
0507 08:05:41.423141 (+ 17672us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:41.429009 10304 service_pool.cc:135] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:39708 dropped due to backpressure. The service queue is full; it has 50 items.
I20240507 08:05:41.393108 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.129658 11258 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.281s
I20240507 08:05:41.399147 11394 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.433511 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.440374 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.118435 11403 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:41.410964 11405 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.452000 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.422485 11188 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.458398 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.458642 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 747) took 1062 ms. Trace:
W20240507 08:05:41.458813 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 1.38043e+06us
I20240507 08:05:41.459270 10338 rpcz_store.cc:276] 0507 08:05:40.396352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.451671 (+1055319us) service_pool.cc:225] Handling call
0507 08:05:41.458551 (+ 6880us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.430858 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 750) took 1032 ms. Trace:
I20240507 08:05:41.409828 11407 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:41.476990 10941 consensus_meta.cc:287] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 1.106s user 0.012s sys 0.004s
I20240507 08:05:41.477941 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.480964 11403 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:41.484730 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 764) took 1397 ms (client timeout 1682 ms). Trace:
W20240507 08:05:41.503638 10341 rpcz_store.cc:269] 0507 08:05:40.087113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.126854 (+ 39741us) service_pool.cc:225] Handling call
0507 08:05:41.484522 (+1357668us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b5400283f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:41.484646 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":188640512}
I20240507 08:05:41.485493 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.407128 11353 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.495728 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.499789 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 115446us
I20240507 08:05:41.456516 11428 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.511576 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1026) took 1210 ms. Trace:
I20240507 08:05:41.512102 10260 rpcz_store.cc:276] 0507 08:05:40.294730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.330174 (+ 35444us) service_pool.cc:225] Handling call
0507 08:05:41.485317 (+1155143us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:41.505296 (+ 19979us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10637056}
W20240507 08:05:41.513171 10258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 744) took 1979 ms (client timeout 2195 ms). Trace:
W20240507 08:05:41.513774 10258 rpcz_store.cc:269] 0507 08:05:39.528604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.051893 (+523289us) service_pool.cc:225] Handling call
0507 08:05:41.495576 (+1443683us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec572e 000000000033cd89 00007fcf95cf46c0 00007fcf95dba3c4 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:41.508017 (+ 12441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2523648}
I20240507 08:05:41.461388 11405 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:41.517719 10155 connection.cc:232] RPC call timeout handler was delayed by -1.3485131780000188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1615s for a response
W20240507 08:05:41.528426 10155 connection.cc:232] RPC call timeout handler was delayed by -1.3178692299999852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
I20240507 08:05:41.528868 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:41.529047 10155 connection.cc:232] RPC call timeout handler was delayed by -1.3145225019999884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20240507 08:05:41.529536 10155 connection.cc:232] RPC call timeout handler was delayed by -1.288213178000035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
W20240507 08:05:41.529933 10155 connection.cc:232] RPC call timeout handler was delayed by -1.2823456360000023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21280000000000002s for a response
I20240507 08:05:41.530169 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.530829 10192 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
W20240507 08:05:41.531412 10155 connection.cc:232] RPC call timeout handler was delayed by -1.0878756970000154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20240507 08:05:41.531664 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:41.531786 10155 connection.cc:232] RPC call timeout handler was delayed by -1.0760646760000441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
W20240507 08:05:41.532007 10184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 750) took 2560 ms (client timeout 2154 ms). Trace:
I20240507 08:05:41.532290 11368 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:05:41.532423 10184 rpcz_store.cc:269] 0507 08:05:38.971563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.403696 (+432133us) service_pool.cc:225] Handling call
0507 08:05:41.531833 (+2128137us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54001be9d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:41.531926 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1983843200}
I20240507 08:05:41.538017 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.494474 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 739) took 1167 ms. Trace:
I20240507 08:05:41.455652 11394 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.460837 11395 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.536335 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.519186 11336 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.525935 11401 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.537477 11384 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.451165 10908 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.548857 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.549722 10258 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:41.550833 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.549839 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.506330 11438 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.488701 10332 rpcz_store.cc:276] 0507 08:05:40.398548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.121001 (+722453us) service_pool.cc:225] Handling call
0507 08:05:41.427428 (+306427us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.532842 11368 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:41.561081 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.551730 10258 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:41.551795 10260 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:41.565172 10192 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:41.566792 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.551673 11370 consensus_meta.cc:287] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.805s user 0.008s sys 0.009s
I20240507 08:05:41.453464 10304 service_pool.cc:138] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:39708 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 784)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 755)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 776)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 749)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 783)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 738)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 782)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 778)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 769)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 752)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 775)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 28)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 786)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 788)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 29)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 780)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 763)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 758)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 777)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 759)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 785)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 30)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 764)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 765)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 757)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 770)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 766)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 771)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 791)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 792)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 789)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 761)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 734)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 779)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 781)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 760)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 762)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 31)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 767)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 768)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 787)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 790)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 772)
I20240507 08:05:41.614285 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:41.614939 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 743) took 1295 ms (client timeout 1661 ms). Trace:
W20240507 08:05:41.615180 10335 rpcz_store.cc:269] 0507 08:05:40.319517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.356830 (+ 37313us) service_pool.cc:225] Handling call
0507 08:05:41.614131 (+1257301us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:41.614895 (+ 764us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9073920}
I20240507 08:05:41.570585 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.570398 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 158442us
I20240507 08:05:41.567672 11395 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.572117 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1037) took 1270 ms. Trace:
I20240507 08:05:41.528439 11405 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.574381 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.621943 10264 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1029) took 1326 ms (client timeout 1711 ms). Trace:
I20240507 08:05:41.558377 11384 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:41.624454 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 91672.7us
I20240507 08:05:41.575475 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.575933 10185 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 735) took 2697 ms (client timeout 2046 ms). Trace:
I20240507 08:05:41.578218 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 738) took 2647 ms. Trace:
I20240507 08:05:41.583593 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 749) took 2642 ms. Trace:
I20240507 08:05:41.584683 11394 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:41.555159 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 75711.5us
I20240507 08:05:41.585325 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.627274 10262 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:41.627203 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.585511 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 737) took 2654 ms. Trace:
I20240507 08:05:41.627144 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.628373 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 755) took 1151 ms. Trace:
I20240507 08:05:41.628023 10190 rpcz_store.cc:276] 0507 08:05:38.930797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.132461 (+1201664us) service_pool.cc:225] Handling call
0507 08:05:40.132721 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.291417 (+158696us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.291447 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.291484 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.291513 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.585288 (+1293775us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54002db9d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.585336 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.585454 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":364771072}
I20240507 08:05:41.628856 10335 rpcz_store.cc:276] 0507 08:05:40.476538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.615774 (+1139236us) service_pool.cc:225] Handling call
0507 08:05:41.628323 (+ 12549us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:41.590059 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 155359us
I20240507 08:05:41.586378 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 742) took 2114 ms. Trace:
I20240507 08:05:41.592301 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.629627 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.629979 10259 rpcz_store.cc:276] 0507 08:05:39.472414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.061558 (+589144us) service_pool.cc:225] Handling call
0507 08:05:40.061815 (+ 257us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.219284 (+157469us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.219315 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.219354 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.219382 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.586159 (+1366777us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400358c54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.586208 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.586324 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":361916160}
I20240507 08:05:41.563777 11438 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.588413 10940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 51) took 2234 ms. Trace:
I20240507 08:05:41.596767 11355 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:41.592124 11431 log.cc:924] Time spent T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
W20240507 08:05:41.592762 11352 consensus_meta.cc:287] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 1.044s user 0.014s sys 0.017s
I20240507 08:05:41.593559 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.508608 11428 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:41.604837 11420 consensus_meta.cc:287] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.664s user 0.000s sys 0.009s
I20240507 08:05:41.614166 11370 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.635334 10940 rpcz_store.cc:276] 0507 08:05:39.404452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.417394 (+ 12942us) service_pool.cc:225] Handling call
0507 08:05:39.417773 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.084445 (+666672us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.094341 (+ 9896us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.100288 (+ 5947us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:40.111616 (+ 11328us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.588140 (+1476524us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b54006d0454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.588197 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.588333 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1494902016}
I20240507 08:05:41.606413 11353 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:41.607966 11247 consensus_meta.cc:287] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.670s user 0.009s sys 0.002s
I20240507 08:05:41.650462 11247 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.650975 11370 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.651679 10189 raft_consensus.cc:2379] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:41.652493 11247 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:41.609978 11368 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:41.619957 10306 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.638870 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.575148 10187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1064) took 2322 ms (client timeout 2175 ms). Trace:
I20240507 08:05:41.655043 10258 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
W20240507 08:05:41.622258 10264 rpcz_store.cc:269] 0507 08:05:40.295439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.322583 (+ 27144us) service_pool.cc:225] Handling call
0507 08:05:41.574225 (+1251642us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:41.621890 (+ 47665us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11388288}
I20240507 08:05:41.626574 10188 rpcz_store.cc:276] 0507 08:05:38.931072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.135104 (+1204032us) service_pool.cc:225] Handling call
0507 08:05:41.576427 (+1441323us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.576656 (+ 229us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:41.576694 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.576749 (+ 55us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:41.576792 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.576870 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.578004 (+ 1134us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54001beed0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:41.578139 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3315706240}
I20240507 08:05:41.631088 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.626758 10183 rpcz_store.cc:276] 0507 08:05:38.940721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.191237 (+1250516us) service_pool.cc:225] Handling call
0507 08:05:40.191598 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.452226 (+260628us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.452257 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.452299 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:40.452329 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.583315 (+1130986us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400260554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.583369 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.583526 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":598330112}
I20240507 08:05:41.658001 10190 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:41.658133 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:41.658743 10188 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.658919 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:41.627851 10262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1033) took 1327 ms (client timeout 1671 ms). Trace:
W20240507 08:05:41.659519 10262 rpcz_store.cc:269] 0507 08:05:40.300453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.322531 (+ 22078us) service_pool.cc:225] Handling call
0507 08:05:41.585134 (+1262603us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:41.627793 (+ 42659us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7937920}
I20240507 08:05:41.659940 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.660531 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.661063 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.632370 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.621047 10267 rpcz_store.cc:276] 0507 08:05:40.301441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.320076 (+1018635us) service_pool.cc:225] Handling call
0507 08:05:41.323967 (+ 3891us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:41.324410 (+ 443us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:41.324451 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:41.324984 (+ 533us) log.cc:842] Serialized 26 byte log entry
0507 08:05:41.405471 (+ 80487us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:41.566484 (+161013us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.571810 (+ 5326us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.571885 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:41.571930 (+ 45us) raft_consensus.cc:1671] finished
0507 08:05:41.571944 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.572066 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":165356,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":2132,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":271,"thread_start_us":234003,"threads_started":2,"wal-append.queue_time_us":192791}
I20240507 08:05:41.626951 10155 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.635730 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.637297 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.648321 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 758) took 1820 ms. Trace:
I20240507 08:05:41.649506 11420 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.542366 10333 rpcz_store.cc:276] 0507 08:05:40.318467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.975212 (+656745us) service_pool.cc:225] Handling call
0507 08:05:41.485539 (+510327us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.661540 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:41.665881 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.661921 10260 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:41.664218 10335 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:41.663265 10155 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.709781 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1038) took 1398 ms. Trace:
I20240507 08:05:41.663975 10259 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:41.710831 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:41.656720 11366 consensus_meta.cc:287] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.701s user 0.006s sys 0.004s
I20240507 08:05:41.711719 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.709565 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.712677 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.712690 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:41.662201 10265 consensus_meta.cc:287] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.606s user 0.007s sys 0.003s
W20240507 08:05:41.724270 11310 consensus_meta.cc:287] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.667s user 0.006s sys 0.003s
I20240507 08:05:41.645815 10184 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:41.724748 11310 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.725522 10190 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:41.725898 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.726351 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:41.726550 10184 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:41.728277 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.729543 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 66383.4us
I20240507 08:05:41.730506 10155 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.736130 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.736804 10259 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:41.710211 10263 rpcz_store.cc:276] 0507 08:05:40.310912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.406643 (+1095731us) service_pool.cc:225] Handling call
0507 08:05:41.406943 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:41.409090 (+ 2147us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:41.409118 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:41.409488 (+ 370us) log.cc:842] Serialized 26 byte log entry
0507 08:05:41.546449 (+136961us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:41.669413 (+122964us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.709473 (+ 40060us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.709538 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:41.709606 (+ 68us) raft_consensus.cc:1671] finished
0507 08:05:41.709618 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.709733 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41485,"prepare.queue_time_us":24644,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":70845,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":9982,"thread_start_us":244052,"threads_started":2,"wal-append.queue_time_us":167163}
I20240507 08:05:41.716686 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.737551 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.738700 10259 raft_consensus.cc:2379] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:41.719434 10183 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
W20240507 08:05:41.626418 10185 rpcz_store.cc:269] 0507 08:05:38.878468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.343907 (+465439us) service_pool.cc:225] Handling call
0507 08:05:41.575750 (+2231843us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b540026dc54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:41.575848 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1452786560}
I20240507 08:05:41.720674 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.720999 10336 raft_consensus.cc:2379] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:41.721283 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.669523 11409 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:41.722033 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.723956 10188 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:41.737833 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 744) took 1418 ms. Trace:
I20240507 08:05:41.746029 10267 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:41.746084 10338 rpcz_store.cc:276] 0507 08:05:40.319770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.716341 (+1396571us) service_pool.cc:225] Handling call
0507 08:05:41.737760 (+ 21419us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.746384 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.717600 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:41.739739 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.740139 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.727128 11384 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.748428 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.749356 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.749325 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.750095 10260 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:41.750339 10259 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:41.743171 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.743304 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:41.743628 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.751900 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1040) took 1354 ms. Trace:
I20240507 08:05:41.752084 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.752233 10263 rpcz_store.cc:276] 0507 08:05:40.397724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.751380 (+1353656us) service_pool.cc:225] Handling call
0507 08:05:41.751599 (+ 219us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.751711 (+ 112us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:41.751726 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.751752 (+ 26us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:41.751769 (+ 17us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.751798 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.751865 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.753043 10183 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:41.754665 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.747438 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.756685 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:41.757416 10183 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:41.713191 11443 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20240507 08:05:41.758294 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.759403 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.708376 11368 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240507 08:05:41.707808 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 52997us
I20240507 08:05:41.766960 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.767253 10232 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:41.764451 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.757481 10263 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20240507 08:05:41.740808 11355 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.736264 11351 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:41.648407 11352 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.774711 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.774960 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.775436 10185 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:41.775624 10940 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:41.775823 11352 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.766860 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.776629 10261 raft_consensus.cc:2379] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:41.776868 10184 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:41.747225 10188 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:41.777767 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.780372 10184 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:41.766458 11319 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.781322 11319 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.781437 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.781774 11319 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.782491 11438 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.769165 10259 raft_consensus.cc:1226] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:41.763227 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 769) took 1390 ms. Trace:
I20240507 08:05:41.771937 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.758050 11269 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:41.655375 10187 rpcz_store.cc:269] 0507 08:05:39.253103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.618699 (+365596us) service_pool.cc:225] Handling call
0507 08:05:41.574952 (+1956253us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54001beed4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:41.575067 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":508505856}
I20240507 08:05:41.785641 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.785831 11420 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.763554 11446 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:41.786280 10187 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:41.786546 11446 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.738446 11366 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.786934 10908 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.787215 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.788130 10187 raft_consensus.cc:2396] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:41.788411 11366 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.776399 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.747444 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.778167 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.768303 10232 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:41.759596 11382 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:41.797052 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1065) took 1499 ms. Trace:
I20240507 08:05:41.797448 10185 rpcz_store.cc:276] 0507 08:05:40.297734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.796174 (+1498440us) service_pool.cc:225] Handling call
0507 08:05:41.796488 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.796664 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:41.796698 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.796741 (+ 43us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:41.796796 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.796857 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.796997 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.798626 11405 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.773471 11355 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.801507 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.801851 11355 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.803617 10307 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:41.788518 10908 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:41.779768 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.780565 11290 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:41.808686 11290 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:41.792521 11269 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.809088 11290 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:41.768352 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:41.781803 10943 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43924 (request call id 53) took 2289 ms (client timeout 1960 ms). Trace:
I20240507 08:05:41.783418 10335 rpcz_store.cc:276] 0507 08:05:40.372418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.714170 (+1341752us) service_pool.cc:225] Handling call
0507 08:05:41.763149 (+ 48979us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.783551 10306 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.813611 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.814507 10306 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:41.770426 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.762796 11368 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.790428 10260 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:41.793327 10188 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:41.793366 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 738) took 1475 ms. Trace:
I20240507 08:05:41.815726 11361 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.817118 11361 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:41.817472 11361 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.794093 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 730) took 2919 ms. Trace:
I20240507 08:05:41.773324 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.786949 11446 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:41.850602 11412 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.854171 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.854862 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.856410 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.783150 11351 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.859840 11351 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.794703 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 760) took 2185 ms. Trace:
I20240507 08:05:41.772424 11336 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:41.861985 11336 raft_consensus.cc:686] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:41.860540 10340 rpcz_store.cc:276] 0507 08:05:39.608871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.259076 (+650205us) service_pool.cc:225] Handling call
0507 08:05:40.290340 (+ 31264us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.391273 (+100933us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.396758 (+ 5485us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.400620 (+ 3862us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:40.404274 (+ 3654us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.794475 (+1390201us) spinlock_profiling.cc:243] Waited 85.1 ms on lock 0x7b540013e554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.794525 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.794646 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":195685248}
I20240507 08:05:41.863410 11336 consensus_queue.cc:237] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:41.864691 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.795495 11271 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:41.795938 11428 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.804809 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:41.805733 10183 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:41.782028 10941 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 52) took 2346 ms. Trace:
I20240507 08:05:41.888677 10941 rpcz_store.cc:276] 0507 08:05:39.444405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.483332 (+ 38927us) service_pool.cc:225] Handling call
0507 08:05:39.588155 (+104823us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:39.644028 (+ 55873us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:05:41.506139 (+1862111us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.747142 (+241003us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.759214 (+ 12072us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:41.763702 (+ 4488us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.769773 (+ 6071us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.778295 (+ 8522us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":49055,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":221858,"threads_started":2,"wal-append.queue_time_us":57957}
I20240507 08:05:41.788727 11447 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.890395 11447 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.890827 11447 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.807811 11309 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.807735 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.811089 10232 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:41.897315 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 86319.7us
I20240507 08:05:41.898202 10232 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:41.815976 11250 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.812181 10939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 70) took 2181 ms. Trace:
I20240507 08:05:41.853233 11412 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:41.816679 11269 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:41.828317 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.815855 11315 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.815575 10338 rpcz_store.cc:276] 0507 08:05:40.318175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.747152 (+1428977us) service_pool.cc:225] Handling call
0507 08:05:41.793312 (+ 46160us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.842489 11353 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.773129 11383 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:41.827104 11384 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.892270 11364 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.908519 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 749) took 1511 ms. Trace:
I20240507 08:05:41.929118 10333 rpcz_store.cc:276] 0507 08:05:40.397007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.727546 (+1330539us) service_pool.cc:225] Handling call
0507 08:05:41.902169 (+174623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.903440 10939 rpcz_store.cc:276] 0507 08:05:39.658203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.877543 (+219340us) service_pool.cc:225] Handling call
0507 08:05:39.877957 (+ 414us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.292127 (+414170us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.296549 (+ 4422us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.303231 (+ 6682us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:40.306237 (+ 3006us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.809511 (+1503274us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b54005113d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.809559 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.811630 (+ 2071us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":924603136}
I20240507 08:05:41.932422 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.912410 11351 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:41.933248 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 752) took 1481 ms. Trace:
I20240507 08:05:41.933609 11351 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.913847 11309 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:41.903999 11412 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.819255 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:41.797060 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.892704 10188 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:41.815927 11368 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.885089 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1035) took 1566 ms. Trace:
I20240507 08:05:41.816257 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.899266 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:41.818575 10186 rpcz_store.cc:276] 0507 08:05:38.872680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:38.872809 (+ 129us) service_pool.cc:225] Handling call
0507 08:05:38.873201 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.220340 (+1347139us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.792246 (+1571906us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54001bf8d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.792402 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2999079808}
I20240507 08:05:41.927732 11364 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:41.963034 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 782) took 1034 ms. Trace:
I20240507 08:05:41.963541 10335 rpcz_store.cc:276] 0507 08:05:40.928503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.812917 (+884414us) service_pool.cc:225] Handling call
0507 08:05:41.962970 (+150053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.962030 10938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43910 (request call id 102) took 1540 ms. Trace:
I20240507 08:05:41.915407 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:41.898614 10185 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
W20240507 08:05:41.939798 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 120636us
I20240507 08:05:41.937219 10307 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:41.782856 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.005278 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.006310 10335 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:42.007273 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 775) took 1520 ms. Trace:
I20240507 08:05:42.007762 10335 rpcz_store.cc:276] 0507 08:05:40.486439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.964549 (+1478110us) service_pool.cc:225] Handling call
0507 08:05:42.007201 (+ 42652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.934115 11351 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:42.011183 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 769) took 1039 ms. Trace:
I20240507 08:05:42.013813 10338 rpcz_store.cc:276] 0507 08:05:40.972032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.915076 (+943044us) service_pool.cc:225] Handling call
0507 08:05:42.011108 (+ 96032us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.934439 11309 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:41.812696 10943 rpcz_store.cc:269] 0507 08:05:39.543509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:39.560741 (+ 17232us) service_pool.cc:225] Handling call
0507 08:05:41.781597 (+2220856us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400512554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:41.781721 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":468618624}
I20240507 08:05:41.925124 11370 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.931406 11310 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:41.903017 11250 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:41.921646 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 759) took 1933 ms. Trace:
I20240507 08:05:41.962303 11364 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:41.998637 11428 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:42.006754 11128 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:41.955785 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 778) took 1034 ms. Trace:
I20240507 08:05:42.078236 10186 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20240507 08:05:42.012810 10184 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:42.076802 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.088713 10338 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:42.089413 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 786) took 1146 ms. Trace:
I20240507 08:05:42.089805 10338 rpcz_store.cc:276] 0507 08:05:40.943143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.014844 (+1071701us) service_pool.cc:225] Handling call
0507 08:05:42.089344 (+ 74500us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.022444 11309 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.024420 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 783) took 1089 ms. Trace:
I20240507 08:05:41.933931 10333 rpcz_store.cc:276] 0507 08:05:40.452565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.932125 (+1479560us) service_pool.cc:225] Handling call
0507 08:05:41.933186 (+ 1061us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.096206 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.059327 11250 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.061651 11428 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.101229 10338 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:42.102452 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 788) took 1035 ms. Trace:
W20240507 08:05:42.102797 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 165132us
W20240507 08:05:42.076910 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 177814us
I20240507 08:05:42.102816 10338 rpcz_store.cc:276] 0507 08:05:41.066687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.095872 (+1029185us) service_pool.cc:225] Handling call
0507 08:05:42.102385 (+ 6513us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.108791 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.103977 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.110025 10233 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:42.110168 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 29) took 1195 ms. Trace:
I20240507 08:05:42.111227 10338 rpcz_store.cc:276] 0507 08:05:40.914714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.108480 (+1193766us) service_pool.cc:225] Handling call
0507 08:05:42.110096 (+ 1616us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.112357 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.113993 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 763) took 1171 ms. Trace:
I20240507 08:05:42.114416 10338 rpcz_store.cc:276] 0507 08:05:40.945498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.112136 (+1166638us) service_pool.cc:225] Handling call
0507 08:05:42.113865 (+ 1729us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:41.963376 11315 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.105367 11383 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.104990 11428 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:42.118003 11427 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.113315 11308 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.133026 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:42.134295 11446 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.003540 10938 rpcz_store.cc:276] 0507 08:05:40.450414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.472023 (+ 21609us) service_pool.cc:225] Handling call
0507 08:05:40.491456 (+ 19433us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.793756 (+302300us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.802764 (+ 9008us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.818582 (+ 15818us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:40.818650 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.934796 (+1116146us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400410354. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:41.934853 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.934983 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":628698496}
I20240507 08:05:42.142437 11355 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.145144 11355 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.136044 11361 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.155323 11319 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.127066 11383 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:42.157289 11319 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.121287 11315 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:42.164430 11315 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.163640 11446 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.153287 11361 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:42.136443 11427 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.138638 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:42.176637 11368 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.174804 11427 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.180155 10939 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:42.187583 10188 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.171769 10906 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:42.231189 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 64408.3us
I20240507 08:05:42.231953 10906 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:42.232306 10188 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:42.232609 11309 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.232738 11428 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.179951 11250 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.187675 11364 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.234015 11395 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:42.236141 11395 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.236505 11395 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:42.241153 10336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 771) took 1865 ms (client timeout 1843 ms). Trace:
W20240507 08:05:42.241887 10336 rpcz_store.cc:269] 0507 08:05:40.373087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.943912 (+570825us) service_pool.cc:225] Handling call
0507 08:05:42.237481 (+1293569us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b540004ced4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:42.237664 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1614899584}
I20240507 08:05:42.243110 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.244839 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 777) took 1442 ms. Trace:
I20240507 08:05:42.244330 11447 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.246889 11309 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.247715 10934 raft_consensus.cc:2379] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:42.245383 10336 rpcz_store.cc:276] 0507 08:05:40.895365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.242776 (+1347411us) service_pool.cc:225] Handling call
0507 08:05:42.244768 (+ 1992us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.251087 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.244657 11383 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.256191 11400 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.262156 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 759) took 1329 ms. Trace:
I20240507 08:05:42.267432 10336 rpcz_store.cc:276] 0507 08:05:40.932641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.250782 (+1318141us) service_pool.cc:225] Handling call
0507 08:05:42.262065 (+ 11283us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.268410 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.269178 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 785) took 1326 ms. Trace:
I20240507 08:05:42.269480 10336 rpcz_store.cc:276] 0507 08:05:40.942816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.268149 (+1325333us) service_pool.cc:225] Handling call
0507 08:05:42.269117 (+ 968us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.270651 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.271276 10336 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:42.271318 11427 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.271831 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 30) took 1354 ms. Trace:
I20240507 08:05:42.272171 10336 rpcz_store.cc:276] 0507 08:05:40.917106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.270399 (+1353293us) service_pool.cc:225] Handling call
0507 08:05:42.271778 (+ 1379us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.243288 11412 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.273380 11427 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.285936 11250 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.266803 11368 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.287164 11428 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.273834 11395 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.287791 11454 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.290025 11454 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:42.290313 11454 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.290549 11412 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.291079 11395 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.291504 11447 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.293916 11364 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.295243 11454 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.273469 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.273610 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.324215 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 764) took 1354 ms. Trace:
W20240507 08:05:42.322454 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 28) took 1831 ms (client timeout 2167 ms). Trace:
I20240507 08:05:42.274739 11290 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.327965 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:42.324839 10336 rpcz_store.cc:276] 0507 08:05:40.969994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.273328 (+1303334us) service_pool.cc:225] Handling call
0507 08:05:42.323916 (+ 50588us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:42.328444 10335 rpcz_store.cc:269] 0507 08:05:40.491074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.008608 (+1517534us) service_pool.cc:225] Handling call
0507 08:05:42.273268 (+264660us) spinlock_profiling.cc:243] Waited 42.1 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:42.322097 (+ 48829us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96827648}
I20240507 08:05:42.334584 11447 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.335390 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.336417 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.337143 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 765) took 1366 ms. Trace:
I20240507 08:05:42.337628 10335 rpcz_store.cc:276] 0507 08:05:40.970454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.334606 (+1364152us) service_pool.cc:225] Handling call
0507 08:05:42.337052 (+ 2446us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.335424 11447 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:42.339092 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.337913 10336 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:42.339938 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 757) took 1408 ms. Trace:
I20240507 08:05:42.340364 10336 rpcz_store.cc:276] 0507 08:05:40.932011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.335802 (+1403791us) service_pool.cc:225] Handling call
0507 08:05:42.339825 (+ 4023us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.340891 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.341218 11315 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.341847 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 766) took 1371 ms. Trace:
I20240507 08:05:42.341661 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.339258 11447 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.343676 10335 rpcz_store.cc:276] 0507 08:05:40.970833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.340452 (+1369619us) service_pool.cc:225] Handling call
0507 08:05:42.341775 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.346267 11454 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.349951 11315 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.350292 11290 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:42.352914 11351 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.353515 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:42.355492 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 527383us
I20240507 08:05:42.355443 11447 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.356114 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.356503 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.357074 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 791) took 1173 ms. Trace:
I20240507 08:05:42.357416 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:42.357520 10335 rpcz_store.cc:276] 0507 08:05:41.183664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.355751 (+1172087us) service_pool.cc:225] Handling call
0507 08:05:42.356993 (+ 1242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.358177 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.427760 11368 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.428398 11368 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.428279 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.428542 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 789) took 1306 ms. Trace:
I20240507 08:05:42.429003 11368 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.429152 10336 rpcz_store.cc:276] 0507 08:05:41.122626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.357794 (+1235168us) service_pool.cc:225] Handling call
0507 08:05:42.428453 (+ 70659us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.429514 10335 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:42.430922 10939 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.431145 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.355639 11351 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.430944 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.433992 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.434729 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.434975 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 761) took 1501 ms. Trace:
I20240507 08:05:42.435622 10335 rpcz_store.cc:276] 0507 08:05:40.933768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.433463 (+1499695us) service_pool.cc:225] Handling call
0507 08:05:42.434906 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.435803 10336 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:42.437269 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.439122 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.440706 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.442195 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.358839 11447 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.443603 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.445115 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.446436 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.447803 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.447770 10336 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:42.450331 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.452432 10335 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.453739 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.059105 10187 rpcz_store.cc:276] 0507 08:05:39.977247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.788691 (+1811444us) service_pool.cc:225] Handling call
0507 08:05:41.804399 (+ 15708us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.840793 (+ 36394us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:41.875053 (+ 34260us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.885270 (+ 10217us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:41.888043 (+ 2773us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.896138 (+ 8095us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.910744 (+ 14606us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.456666 11400 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:42.457162 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.457396 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.458074 11400 raft_consensus.cc:3032] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:42.458560 11400 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:42.459702 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.459935 11400 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.460716 10186 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
W20240507 08:05:42.457269 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 100065us
I20240507 08:05:42.462378 10155 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.463619 11446 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:42.464073 11446 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:42.464563 11446 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:42.461557 11400 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.469007 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:41.957640 10265 rpcz_store.cc:276] 0507 08:05:40.300953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.476992 (+176039us) service_pool.cc:225] Handling call
0507 08:05:40.549097 (+ 72105us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.588464 (+ 39367us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:05:41.724658 (+1136194us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.780069 (+ 55411us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.780126 (+ 57us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:41.790681 (+ 10555us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.805442 (+ 14761us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.867310 (+ 61868us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5168,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":95,"thread_start_us":42007,"threads_started":1,"wal-append.queue_time_us":45480}
I20240507 08:05:42.467597 11368 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.476310 10187 raft_consensus.cc:1226] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.471258 10155 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.453474 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.482252 10190 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.482430 10155 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:42.483681 11355 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.484552 11355 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:42.484539 10155 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.485108 11355 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.450320 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.485870 11355 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:42.487315 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1039) took 2166 ms. Trace:
I20240507 08:05:42.488163 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 779) took 1589 ms. Trace:
I20240507 08:05:42.488371 11356 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.488622 10336 rpcz_store.cc:276] 0507 08:05:40.899158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.486757 (+1587599us) service_pool.cc:225] Handling call
0507 08:05:42.487362 (+ 605us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.487785 (+ 423us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.487818 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.487880 (+ 62us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:42.487909 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.487972 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.488096 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.487777 10260 rpcz_store.cc:276] 0507 08:05:40.321131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.750602 (+1429471us) service_pool.cc:225] Handling call
0507 08:05:41.750810 (+ 208us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.815259 (+ 64449us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.486770 (+671511us) spinlock_profiling.cc:243] Waited 39.5 ms on lock 0x7b5400359654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:42.487066 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":90782464}
I20240507 08:05:42.491408 11368 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.492200 10155 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.494928 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 781) took 1581 ms. Trace:
I20240507 08:05:42.495355 11368 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.495332 10336 rpcz_store.cc:276] 0507 08:05:40.913237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.493919 (+1580682us) service_pool.cc:225] Handling call
0507 08:05:42.494319 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.494574 (+ 255us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.494600 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.494661 (+ 61us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:42.494690 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.494750 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.494859 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.498198 10155 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.499348 11400 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.499845 11400 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.500345 11400 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.500172 10155 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.501469 11356 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.501744 10155 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.501953 11356 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.502743 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 734) took 2137 ms. Trace:
I20240507 08:05:42.503108 11353 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:42.502446 11356 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.510176 11353 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.510689 11353 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:42.506211 10335 rpcz_store.cc:276] 0507 08:05:40.316258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.451477 (+2135219us) service_pool.cc:225] Handling call
0507 08:05:42.452030 (+ 553us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.452881 (+ 851us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.453883 (+ 1002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.511284 10190 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:42.512799 10335 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:42.514557 10155 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.517875 11310 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.518507 11310 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.518837 10155 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.519002 11310 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.521082 11446 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:42.521711 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 55144.8us
I20240507 08:05:42.520397 10155 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.077675 10340 rpcz_store.cc:276] 0507 08:05:40.897295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.863254 (+965959us) service_pool.cc:225] Handling call
0507 08:05:41.931705 (+ 68451us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.524519 11446 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.525156 11446 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:42.525847 11446 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.526221 11446 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.527241 11446 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.528304 11447 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.516779 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.529245 11447 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:42.530225 10334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 742) took 2210 ms (client timeout 1729 ms). Trace:
W20240507 08:05:42.530652 10334 rpcz_store.cc:269] 0507 08:05:40.319273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.357902 (+ 38629us) service_pool.cc:225] Handling call
0507 08:05:42.516383 (+2158481us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:42.530128 (+ 13745us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33216256}
I20240507 08:05:42.532820 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 767) took 1561 ms. Trace:
I20240507 08:05:42.532970 11447 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.533421 10334 rpcz_store.cc:276] 0507 08:05:40.971310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.531528 (+1560218us) service_pool.cc:225] Handling call
0507 08:05:42.531961 (+ 433us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.532351 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.532392 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.532477 (+ 85us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:42.532522 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.532601 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.532746 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.540081 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 768) took 1568 ms. Trace:
I20240507 08:05:42.540961 10340 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:42.540556 11446 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.541884 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 31) took 1582 ms. Trace:
I20240507 08:05:42.545189 10340 rpcz_store.cc:276] 0507 08:05:40.959488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.524727 (+1565239us) service_pool.cc:225] Handling call
0507 08:05:42.525342 (+ 615us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.541495 (+ 16153us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.541789 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.547291 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 787) took 1555 ms. Trace:
I20240507 08:05:42.547624 10340 rpcz_store.cc:276] 0507 08:05:40.991634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.546109 (+1554475us) service_pool.cc:225] Handling call
0507 08:05:42.546571 (+ 462us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.546943 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.546971 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.547031 (+ 60us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:42.547061 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.547118 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.547241 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.548432 11310 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.551190 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.552331 10190 raft_consensus.cc:2379] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:42.552769 11356 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:42.553869 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48147.2us
I20240507 08:05:42.554930 11356 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.544342 11353 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.556694 11353 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.558984 10090 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59948 (request call id 160) took 5560 ms. Trace:
I20240507 08:05:42.560335 11355 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.572064 11400 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.562288 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:42.579174 10935 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
W20240507 08:05:42.579674 10935 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43936 (request call id 74) took 1597 ms (client timeout 2095 ms). Trace:
W20240507 08:05:42.580062 10935 rpcz_store.cc:269] 0507 08:05:40.982493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.983353 (+ 860us) service_pool.cc:225] Handling call
0507 08:05:42.562007 (+1578654us) spinlock_profiling.cc:243] Waited 9.63 ms on lock 0x7b58002f2e98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec572e 000000000033cd89 00007fcf95cf46c0 00007fcf95dba3c4 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:42.579610 (+ 17603us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22157568}
I20240507 08:05:42.564098 11310 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:42.565874 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 45639.1us
I20240507 08:05:42.549090 11446 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.568439 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:41.664376 10192 rpcz_store.cc:276] 0507 08:05:39.828136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.574142 (+1746006us) service_pool.cc:225] Handling call
0507 08:05:41.574450 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.574633 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:41.574926 (+ 293us) log.cc:842] Serialized 14 byte log entry
0507 08:05:41.647877 (+ 72951us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:41.647926 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:41.647963 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:41.648032 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:41.648262 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":14203,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":5947,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":11203,"thread_start_us":72482,"threads_started":1}
I20240507 08:05:42.595822 10939 raft_consensus.cc:2379] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
W20240507 08:05:42.569713 10934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43936 (request call id 75) took 1583 ms (client timeout 1968 ms). Trace:
W20240507 08:05:42.596731 10934 rpcz_store.cc:269] 0507 08:05:40.982842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.991704 (+ 8862us) service_pool.cc:225] Handling call
0507 08:05:42.132838 (+1141134us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b58002f2e98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:42.561336 (+428498us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54006d15d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:42.566102 (+ 4766us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273797760}
I20240507 08:05:42.566128 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 790) took 1427 ms. Trace:
I20240507 08:05:42.597988 10340 rpcz_store.cc:276] 0507 08:05:41.124947 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.548615 (+1423668us) service_pool.cc:225] Handling call
0507 08:05:42.551229 (+ 2614us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.551629 (+ 400us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.551667 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.551735 (+ 68us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:42.551807 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.551880 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.552044 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.599228 10158 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.599972 10340 raft_consensus.cc:1226] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.600601 11446 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.574060 11355 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.571287 10090 rpcz_store.cc:276] 0507 08:05:36.997853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:36.998004 (+ 151us) service_pool.cc:225] Handling call
0507 08:05:42.558675 (+5560671us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:37.914022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:37.914527 (+ 505us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:37.914563 (+ 36us) write_op.cc:435] Acquired schema lock
0507 08:05:37.914580 (+ 17us) tablet.cc:660] Decoding operations
0507 08:05:37.915165 (+ 585us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:37.915222 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:37.915259 (+ 37us) tablet.cc:683] Acquiring locks for 4 operations
0507 08:05:37.915640 (+ 381us) tablet.cc:699] Row locks acquired
0507 08:05:37.915655 (+ 15us) write_op.cc:260] PREPARE: finished
0507 08:05:37.915792 (+ 137us) write_op.cc:270] Start()
0507 08:05:37.915892 (+ 100us) write_op.cc:276] Timestamp: P: 1715069137915745 usec, L: 0
0507 08:05:37.915912 (+ 20us) op_driver.cc:331] REPLICATION: starting
0507 08:05:37.916503 (+ 591us) log.cc:842] Serialized 1705 byte log entry
0507 08:05:38.733153 (+816650us) op_driver.cc:444] REPLICATION: finished
0507 08:05:38.866003 (+132850us) write_op.cc:301] APPLY: starting
0507 08:05:38.888566 (+ 22563us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:38.944170 (+ 55604us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:38.965482 (+ 21312us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:38.974090 (+ 8608us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:38.974335 (+ 245us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0507 08:05:38.991115 (+ 16780us) write_op.cc:312] APPLY: finished
0507 08:05:39.038954 (+ 47839us) log.cc:842] Serialized 40 byte log entry
0507 08:05:39.039310 (+ 356us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:39.039423 (+ 113us) write_op.cc:454] Released schema lock
0507 08:05:39.039464 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:39.090305 (+ 50841us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:39.091150 (+ 845us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2732,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":82236,"apply.run_cpu_time_us":3735,"apply.run_wall_time_us":295749,"num_ops":4,"prepare.queue_time_us":510395,"prepare.run_cpu_time_us":11241,"prepare.run_wall_time_us":317408,"raft.queue_time_us":165309,"raft.run_cpu_time_us":9078,"raft.run_wall_time_us":26566,"replication_time_us":817145,"thread_start_us":641489,"threads_started":4,"wal-append.queue_time_us":551893,"wal-append.run_cpu_time_us":71089,"wal-append.run_wall_time_us":1583983}]]}
I20240507 08:05:42.584715 11400 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.585239 10155 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.592939 10935 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.604204 10155 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.606177 11355 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.589191 10088 catalog_manager.cc:5484] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:42.600688 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 772) took 1197 ms. Trace:
I20240507 08:05:42.540500 10334 rpcz_store.cc:276] 0507 08:05:40.971708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.538432 (+1566724us) service_pool.cc:225] Handling call
0507 08:05:42.539051 (+ 619us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.539570 (+ 519us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.539621 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.539698 (+ 77us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:42.539740 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.539819 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.540004 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.612453 10334 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.613658 10340 rpcz_store.cc:276] 0507 08:05:41.402974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.599141 (+1196167us) service_pool.cc:225] Handling call
0507 08:05:42.599629 (+ 488us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.600397 (+ 768us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.600623 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.617739 11355 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.591655 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 762) took 1652 ms. Trace:
I20240507 08:05:42.618672 10335 rpcz_store.cc:276] 0507 08:05:40.939274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.511607 (+1572333us) service_pool.cc:225] Handling call
0507 08:05:42.512060 (+ 453us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.512502 (+ 442us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:05:42.563853 (+ 51351us) log.cc:842] Serialized 14 byte log entry
0507 08:05:42.591259 (+ 27406us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.591314 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:42.591353 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.591420 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.591567 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":178,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":183,"thread_start_us":26470,"threads_started":1,"wal-append.queue_time_us":26907}
I20240507 08:05:42.605126 11400 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.100576 10332 rpcz_store.cc:276] 0507 08:05:40.933439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.758328 (+824889us) service_pool.cc:225] Handling call
0507 08:05:41.796884 (+ 38556us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:42.023024 (+226140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13356928}
I20240507 08:05:42.620678 11310 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:42.621763 11400 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:42.622341 11400 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.622501 11310 raft_consensus.cc:3032] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:42.623159 11310 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:42.624827 11310 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.115676 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.633757 11310 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:42.637499 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 741) took 2318 ms (client timeout 2154 ms). Trace:
W20240507 08:05:42.645481 10341 rpcz_store.cc:269] 0507 08:05:40.319045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.637145 (+2318100us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:42.637326 (+ 181us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:42.646508 10338 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 758) took 1714 ms (client timeout 1823 ms). Trace:
W20240507 08:05:42.647439 10338 rpcz_store.cc:269] 0507 08:05:40.932369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.115378 (+1183009us) service_pool.cc:225] Handling call
0507 08:05:42.646269 (+530891us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:42.609059 10088 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:42.649080 11383 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.660503 11400 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.677240 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 765) took 2416 ms. Trace:
I20240507 08:05:42.677752 10337 rpcz_store.cc:276] 0507 08:05:40.260640 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.307145 (+ 46505us) service_pool.cc:225] Handling call
0507 08:05:40.307433 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:40.370908 (+ 63475us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:40.370956 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:40.371012 (+ 56us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:40.371054 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.676742 (+2305688us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b5400049f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:42.676811 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.677048 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":145534208}
W20240507 08:05:42.691243 10231 leader_election.cc:334] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.661s (SENT)
I20240507 08:05:42.691834 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:42.693161 10184 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.697033 10233 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:42.701179 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:42.702003 10260 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:42.703794 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:42.705736 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:42.707774 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:42.708593 10260 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:42.709776 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.711622 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.713682 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:42.715425 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.719583 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.723656 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.724824 10260 raft_consensus.cc:2379] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:42.726013 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.726505 11180 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.726915 10262 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:42.728559 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.729333 10262 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.731319 11250 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.731714 11250 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:42.732066 11250 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.735447 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:42.737216 10262 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:42.739852 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.741655 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.742368 10258 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.749912 10258 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:42.751969 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.752642 10306 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:42.752957 10258 raft_consensus.cc:2379] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:42.754624 10184 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.756486 10188 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.756747 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.757684 10188 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.758071 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.759423 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.760107 10306 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:42.760879 11271 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.761657 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.762014 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.762382 10260 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.762852 10186 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.770754 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.772136 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.772758 11258 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.773185 11258 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:42.773568 11258 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.769330 11315 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20240507 08:05:42.778064 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.761341 11271 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:42.781810 11271 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.785777 10186 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:42.785996 11247 consensus_queue.cc:1046] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:42.787163 11247 raft_consensus.cc:3032] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:42.787611 11247 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:42.788610 11247 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.779124 11238 consensus_queue.cc:1046] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240507 08:05:42.779817 10231 leader_election.cc:334] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.823s (SENT)
I20240507 08:05:42.780323 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.791388 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:42.781371 11315 raft_consensus.cc:3032] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:42.794335 11315 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:42.795361 11315 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.798662 11315 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.798817 11377 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.799547 10260 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:42.790789 11238 raft_consensus.cc:3032] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:42.801445 11238 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:42.802747 11238 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.804124 11238 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:42.804306 10156 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.805200 11368 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.806169 10907 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.807569 10907 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.810351 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:42.811447 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 780) took 1908 ms (client timeout 1814 ms). Trace:
W20240507 08:05:42.811830 10333 rpcz_store.cc:269] 0507 08:05:40.903095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.111984 (+1208889us) service_pool.cc:225] Handling call
0507 08:05:42.811361 (+699377us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:42.812465 10231 leader_election.cc:334] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.893s (SENT)
I20240507 08:05:42.812788 11454 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.813055 10231 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:42.813325 11454 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:42.813860 11454 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.824411 11331 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.823841 11247 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.821944 11447 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.827476 11447 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:42.827556 10339 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 774) took 2341 ms (client timeout 1614 ms). Trace:
I20240507 08:05:42.823251 10086 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 61) took 5129 ms. Trace:
W20240507 08:05:42.829596 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 66991.1us
I20240507 08:05:42.828076 11447 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:42.828545 10339 rpcz_store.cc:269] 0507 08:05:40.486088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.785115 (+299027us) service_pool.cc:225] Handling call
0507 08:05:42.827218 (+2042103us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b5400284954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:42.827373 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206198144}
I20240507 08:05:42.819495 11250 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.832217 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:42.847651 11258 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:42.857724 11250 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:42.850016 11271 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.829522 10086 rpcz_store.cc:276] 0507 08:05:37.616321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.616580 (+ 259us) service_pool.cc:225] Handling call
0507 08:05:42.745532 (+5128952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:41.783435 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:42.022390 (+238955us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:42.024169 (+ 1779us) write_op.cc:435] Acquired schema lock
0507 08:05:42.030710 (+ 6541us) tablet.cc:660] Decoding operations
0507 08:05:42.104647 (+ 73937us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:42.113203 (+ 8556us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:42.117624 (+ 4421us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:05:42.126644 (+ 9020us) tablet.cc:699] Row locks acquired
0507 08:05:42.130083 (+ 3439us) write_op.cc:260] PREPARE: finished
0507 08:05:42.153623 (+ 23540us) write_op.cc:270] Start()
0507 08:05:42.153761 (+ 138us) write_op.cc:276] Timestamp: P: 1715069142140180 usec, L: 0
0507 08:05:42.153782 (+ 21us) op_driver.cc:331] REPLICATION: starting
0507 08:05:42.154424 (+ 642us) log.cc:842] Serialized 598 byte log entry
0507 08:05:42.294817 (+140393us) op_driver.cc:444] REPLICATION: finished
0507 08:05:42.296180 (+ 1363us) write_op.cc:301] APPLY: starting
0507 08:05:42.296262 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:42.296355 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:42.296481 (+ 126us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:42.296654 (+ 173us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:42.296762 (+ 108us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:05:42.296792 (+ 30us) write_op.cc:312] APPLY: finished
0507 08:05:42.296969 (+ 177us) log.cc:842] Serialized 22 byte log entry
0507 08:05:42.297077 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:42.297133 (+ 56us) write_op.cc:454] Released schema lock
0507 08:05:42.297158 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:42.342373 (+ 45215us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:42.342445 (+ 72us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2208417,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1077,"apply.run_cpu_time_us":2189,"apply.run_wall_time_us":46672,"num_ops":1,"prepare.queue_time_us":164472,"prepare.run_cpu_time_us":16822,"prepare.run_wall_time_us":413230,"raft.queue_time_us":44271,"raft.run_cpu_time_us":1254,"raft.run_wall_time_us":1301,"replication_time_us":140950,"thread_start_us":125683,"threads_started":4,"wal-append.queue_time_us":91713}]]}
I20240507 08:05:42.865298 11271 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.866094 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.869469 10306 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:42.871281 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 757619us
I20240507 08:05:42.876298 10232 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.876345 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.876307 11238 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:42.878861 11238 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:42.879359 11238 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.881050 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1043) took 2396 ms. Trace:
I20240507 08:05:42.881561 10261 rpcz_store.cc:276] 0507 08:05:40.484620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.795673 (+311053us) service_pool.cc:225] Handling call
0507 08:05:42.880565 (+2084892us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54001d2254. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:42.880804 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2136538880}
I20240507 08:05:42.882185 11377 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:42.882759 11377 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:42.886219 11331 consensus_queue.cc:1046] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:42.887614 11331 consensus_queue.cc:1046] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:42.888510 11258 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:42.888684 11331 consensus_queue.cc:1046] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:42.889663 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.889458 11331 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:42.891537 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.872975 11454 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.891865 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.883962 11377 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:42.892781 11464 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20240507 08:05:42.894165 10186 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:42.891644 11250 raft_consensus.cc:3032] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:42.895223 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.894172 11331 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:42.896301 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.901959 10333 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.901191 11331 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.904388 11331 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.906765 10186 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:42.907109 11380 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:42.907517 11380 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:42.908432 11380 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.909670 11250 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:42.910647 11250 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.911598 11380 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:42.917121 11407 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.898895 10339 raft_consensus.cc:2379] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:42.920959 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.922462 10333 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:42.924435 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.911983 11250 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.926136 10306 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:42.928638 11447 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:42.929885 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.930467 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1089) took 1269 ms. Trace:
I20240507 08:05:42.930827 10185 rpcz_store.cc:276] 0507 08:05:41.660761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.088491 (+427730us) service_pool.cc:225] Handling call
0507 08:05:42.929674 (+841183us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.930012 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:42.930049 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.930097 (+ 48us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:42.930126 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.930184 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:42.930316 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:42.933414 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1068) took 2042 ms (client timeout 1530 ms). Trace:
W20240507 08:05:42.933804 10189 rpcz_store.cc:269] 0507 08:05:40.895430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.064814 (+169384us) service_pool.cc:225] Handling call
0507 08:05:42.933013 (+1868199us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b5400261954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:42.933219 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1347591424}
I20240507 08:05:42.934866 11315 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.935798 10188 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.936096 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:42.936621 10231 leader_election.cc:334] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 2.025s (SENT)
I20240507 08:05:42.937103 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:42.938908 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.938966 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:42.939918 10189 raft_consensus.cc:2379] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:42.940544 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.942008 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.942623 11380 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:42.943470 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.945967 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.946630 10339 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.946694 10187 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:42.954453 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.955884 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.957638 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:42.957955 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.959240 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.960139 10338 raft_consensus.cc:2396] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20240507 08:05:42.959937 11315 consensus_queue.cc:1046] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:42.957229 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.958741 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.963212 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.963517 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.964653 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.965098 11447 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.963246 10333 raft_consensus.cc:2379] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:42.962256 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.966909 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.967545 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.967922 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.968176 10341 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:42.966218 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.969069 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.971117 10189 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:42.971566 11309 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:42.972340 11309 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:42.973301 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.973533 11309 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:42.974164 10340 raft_consensus.cc:2354] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4.
I20240507 08:05:42.971611 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.976131 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:42.976822 10332 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:42.945434 11454 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:42.974872 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.981153 10339 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:42.982445 10187 raft_consensus.cc:1261] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:42.982167 10191 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:42.945995 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:42.987634 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:42.986053 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.004040 10185 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.007066 10190 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.018168 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.019279 11409 raft_consensus.cc:3032] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.019680 11409 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.020555 10338 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:43.020625 11409 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.022529 11409 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:43.023159 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44864.6us
W20240507 08:05:43.023447 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44142.6us
I20240507 08:05:43.025791 10186 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:43.030992 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.033407 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 760) took 2100 ms. Trace:
I20240507 08:05:43.034117 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.034826 10265 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.036248 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.037501 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.038089 10260 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.038023 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.027143 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.038000 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.037992 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:43.041260 11400 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:43.042143 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.033815 10336 rpcz_store.cc:276] 0507 08:05:40.932927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:42.496457 (+1563530us) service_pool.cc:225] Handling call
0507 08:05:42.828344 (+331887us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.832897 (+ 4553us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:42.850687 (+ 17790us) log.cc:842] Serialized 14 byte log entry
0507 08:05:42.897883 (+ 47196us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.897928 (+ 45us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:42.898941 (+ 1013us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.899005 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:43.033243 (+134238us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400284950. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:43.033336 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28672,"prepare.queue_time_us":51723,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":13566,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4628,"spinlock_wait_cycles":762466048,"thread_start_us":31154,"threads_started":2,"wal-append.queue_time_us":43237}
I20240507 08:05:43.048447 10337 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.049206 10155 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.052052 10155 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.053476 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.054975 11400 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:43.055244 11407 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.055418 11400 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:43.055585 11356 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.055888 11400 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.054656 10155 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.060590 10332 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.061374 10341 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.062053 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:43.063869 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.064652 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:43.049281 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.069314 11427 consensus_queue.cc:1046] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.070221 10934 raft_consensus.cc:1226] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:43.070487 11427 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.071053 11427 raft_consensus.cc:686] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:05:43.072422 10231 leader_election.cc:334] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 2.163s (SENT)
I20240507 08:05:43.072099 11427 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.072587 11331 consensus_queue.cc:1046] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:43.072889 10231 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.073985 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.075985 11401 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:43.084681 11362 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.078356 11319 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.076685 11268 consensus_queue.cc:1046] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240507 08:05:43.075488 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:43.073562 11331 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.100288 11331 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.101243 11331 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.101821 11315 consensus_queue.cc:1046] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.102468 11331 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.052078 11238 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.028185 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.086306 10260 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:43.086385 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.079949 10939 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.081966 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:43.080595 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.084407 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.076663 11428 raft_consensus.cc:3032] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.109426 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.109810 11428 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.110006 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.111191 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.110810 11428 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.112195 11428 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.112244 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.114825 11369 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.115845 11369 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.094975 10341 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.116986 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.086102 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.117449 10338 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.117913 10336 raft_consensus.cc:2354] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4.
I20240507 08:05:43.117054 11238 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:43.118899 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.119206 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.119997 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.120589 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.121018 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.113384 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.121999 10341 raft_consensus.cc:2379] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:43.123405 10333 raft_consensus.cc:2379] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:43.123625 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.123126 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.124687 10339 raft_consensus.cc:3032] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.125038 10339 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.110016 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.092567 11401 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:43.129474 10231 leader_election.cc:334] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.729s (SENT)
I20240507 08:05:43.128190 10155 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.131806 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 756) took 3423 ms. Trace:
I20240507 08:05:43.126703 11368 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.133199 11368 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.133791 10266 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.134263 11368 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.136816 10334 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:43.138381 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.138633 11401 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.139065 11315 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.139436 11315 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.140280 11315 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.145332 10231 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.132308 10191 rpcz_store.cc:276] 0507 08:05:39.708507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:40.508748 (+800241us) service_pool.cc:225] Handling call
0507 08:05:42.977048 (+2468300us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:05:42.978947 (+ 1899us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:42.981907 (+ 2960us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:05:43.113714 (+131807us) log.cc:842] Serialized 14 byte log entry
0507 08:05:43.131419 (+ 17705us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:43.131487 (+ 68us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:43.131543 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:43.131611 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:43.131721 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":52807552,"thread_start_us":17036,"threads_started":1,"wal-append.queue_time_us":17506}
I20240507 08:05:43.117082 11369 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.141489 11315 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.151782 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:43.154476 10339 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.150661 11238 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240507 08:05:43.167891 11362 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.168573 10935 raft_consensus.cc:1226] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.155606 11369 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.151934 11310 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.153750 10934 raft_consensus.cc:2379] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:43.169195 10231 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.169592 10259 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.165020 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.173601 10339 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.169667 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.177116 10259 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.152915 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.180202 11310 raft_consensus.cc:686] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.183033 11358 raft_consensus.cc:3032] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:43.183259 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1070) took 1528 ms. Trace:
I20240507 08:05:43.182952 11310 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.183531 11358 raft_consensus.cc:729] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.184444 11358 consensus_queue.cc:260] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.183676 10183 rpcz_store.cc:276] 0507 08:05:41.655009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:41.887139 (+232130us) service_pool.cc:225] Handling call
0507 08:05:41.887438 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:41.919983 (+ 32545us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:41.935271 (+ 15288us) log.cc:842] Serialized 14 byte log entry
0507 08:05:42.102547 (+167276us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:42.102594 (+ 47us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:42.102631 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:42.591537 (+488906us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b180029ee40. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8decab06 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089
0507 08:05:43.172526 (+580989us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b540026be54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:43.172575 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:43.183065 (+ 10490us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37241,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":103,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":5268,"spinlock_wait_cycles":80638464,"thread_start_us":166880,"threads_started":1,"wal-append.queue_time_us":219605,"wal-append.run_cpu_time_us":1351,"wal-append.run_wall_time_us":1013895}
I20240507 08:05:43.185806 10263 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.186115 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:43.180068 11369 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.186857 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.187405 11369 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.188019 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.189378 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.190750 10261 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.190573 11255 consensus_queue.cc:1046] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.191159 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.191614 11255 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.191937 11255 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.192761 11255 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.193291 10266 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.193902 11351 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.194032 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:43.187335 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.195408 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:43.197934 10258 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.201678 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.173233 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.177966 11427 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.203588 10341 raft_consensus.cc:2379] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:43.205006 10260 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
W20240507 08:05:43.248790 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 44629.7us
W20240507 08:05:43.249709 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 54407.5us
I20240507 08:05:43.252599 10155 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.202869 11361 raft_consensus.cc:3032] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.257143 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:43.202507 11418 consensus_queue.cc:1046] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240507 08:05:43.260255 10075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 88759.5us
I20240507 08:05:43.263046 11336 consensus_queue.cc:1046] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20240507 08:05:43.264190 11336 raft_consensus.cc:3032] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.264528 11336 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.265839 11356 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
W20240507 08:05:43.266264 11475 log.cc:924] Time spent T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240507 08:05:43.266434 11356 raft_consensus.cc:686] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.267714 11255 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.267532 11356 consensus_queue.cc:237] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.268962 10155 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.250943 11238 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:43.270926 11460 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.271407 11460 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.271675 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.271955 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.272312 11460 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.276196 10262 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:43.266396 10086 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 2 to 3, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:43.179261 11400 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:43.287348 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 118304us
I20240507 08:05:43.290405 10339 raft_consensus.cc:1261] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:43.252306 11271 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.255338 11358 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:43.258810 11361 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:05:43.251636 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47818.8us
I20240507 08:05:43.298576 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.294030 11400 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:43.301476 11309 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:43.340051 10192 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.302146 10938 raft_consensus.cc:1226] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.304819 10188 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.281527 11447 raft_consensus.cc:3032] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:43.305271 10086 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to <none>, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:43.307720 10265 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
W20240507 08:05:43.308347 10304 connection.cc:232] RPC call timeout handler was delayed by -1.3451136869999232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
W20240507 08:05:43.345254 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 76399.7us
I20240507 08:05:43.343047 10341 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.284379 11315 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.339607 10231 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.282747 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.348304 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:43.297462 11271 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.357437 10183 raft_consensus.cc:1226] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.359386 11180 raft_consensus.cc:3032] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.265301 11336 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.372933 10192 raft_consensus.cc:1226] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:43.373642 11336 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:43.375000 10304 connection.cc:232] RPC call timeout handler was delayed by -1.2641136869999627s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20240507 08:05:43.375387 10304 connection.cc:232] RPC call timeout handler was delayed by -1.2281136869999614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17110000000000003s for a response
I20240507 08:05:43.378772 10340 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.383052 11271 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.377336 10304 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.369411 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.367869 11180 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.369411 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:43.369828 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:43.384879 11309 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.345284 11447 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
W20240507 08:05:43.405112 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 107121us
I20240507 08:05:43.350055 11319 consensus_queue.cc:1046] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20240507 08:05:43.408059 11452 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20240507 08:05:43.409678 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.410778 11238 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.411257 11401 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.404564 11369 consensus_queue.cc:1046] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20240507 08:05:43.416766 10192 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.413328 11401 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:43.419495 10263 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:43.386266 11351 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.428486 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.431041 10088 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59936 (request call id 155) took 5946 ms. Trace:
I20240507 08:05:43.443318 11361 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.429075 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.466295 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.428099 10341 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.467016 11361 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.434432 11271 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.448052 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.430233 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.424736 11447 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.438978 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.434340 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.476934 10334 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:43.431485 11319 raft_consensus.cc:3032] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.441046 10185 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.477952 11319 raft_consensus.cc:729] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.462682 10187 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:43.429890 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.479070 11319 consensus_queue.cc:260] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.436372 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.480396 11319 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:43.481062 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 72058.3us
I20240507 08:05:43.446221 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.481871 10259 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.482023 10231 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.458354 10090 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:43.483006 11366 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:43.480438 11447 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:43.430701 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:43.429216 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 52015.1us
I20240507 08:05:43.394217 11315 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:05:43.495838 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 63171.8us
I20240507 08:05:43.484208 10090 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 1, leader changed from <none> to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:43.498287 10183 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.500140 10189 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.496814 10156 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.372857 10906 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.504372 10259 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.481173 11351 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.505465 10155 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:43.506076 10907 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.506392 11356 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.508664 10191 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.509683 11478 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.504333 11362 consensus_queue.cc:1046] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240507 08:05:43.510286 11478 raft_consensus.cc:686] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.511193 11478 consensus_queue.cc:237] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.512945 10156 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.515170 10304 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.514963 11180 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.516170 11180 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.517091 11383 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.518446 10156 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.518718 10332 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:43.519747 10156 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.520388 11368 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.520956 11368 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.521175 11362 raft_consensus.cc:3032] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.521626 11362 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.521689 11309 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:43.521886 11368 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.522619 10231 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.536880 10090 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:43.539340 11263 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.539535 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:43.539793 11470 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.466157 10088 rpcz_store.cc:276] 0507 08:05:37.484256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:37.484590 (+ 334us) service_pool.cc:225] Handling call
0507 08:05:43.430986 (+5946396us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:43.118819 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:43.119381 (+ 562us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:43.119428 (+ 47us) write_op.cc:435] Acquired schema lock
0507 08:05:43.119446 (+ 18us) tablet.cc:660] Decoding operations
0507 08:05:43.119883 (+ 437us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:43.119979 (+ 96us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:43.120009 (+ 30us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:05:43.120446 (+ 437us) tablet.cc:699] Row locks acquired
0507 08:05:43.120463 (+ 17us) write_op.cc:260] PREPARE: finished
0507 08:05:43.120614 (+ 151us) write_op.cc:270] Start()
0507 08:05:43.120757 (+ 143us) write_op.cc:276] Timestamp: P: 1715069143120559 usec, L: 0
0507 08:05:43.120792 (+ 35us) op_driver.cc:331] REPLICATION: starting
0507 08:05:43.141814 (+ 21022us) log.cc:842] Serialized 928 byte log entry
0507 08:05:43.273219 (+131405us) op_driver.cc:444] REPLICATION: finished
0507 08:05:43.289263 (+ 16044us) write_op.cc:301] APPLY: starting
0507 08:05:43.289395 (+ 132us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:43.289560 (+ 165us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:43.289751 (+ 191us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:43.290385 (+ 634us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:43.290689 (+ 304us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0507 08:05:43.290776 (+ 87us) write_op.cc:312] APPLY: finished
0507 08:05:43.306235 (+ 15459us) log.cc:842] Serialized 28 byte log entry
0507 08:05:43.346466 (+ 40231us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:43.346562 (+ 96us) write_op.cc:454] Released schema lock
0507 08:05:43.346621 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:43.373224 (+ 26603us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:43.373253 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":15729,"apply.run_cpu_time_us":3115,"apply.run_wall_time_us":84283,"num_ops":2,"prepare.queue_time_us":447332,"prepare.run_cpu_time_us":3228,"prepare.run_wall_time_us":23301,"replication_time_us":152335,"thread_start_us":15244,"threads_started":1}]]}
I20240507 08:05:43.540131 11255 consensus_queue.cc:1046] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:43.556254 11263 raft_consensus.cc:3032] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:43.522516 11362 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.556659 11263 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.557395 11362 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.557504 11263 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.558869 11263 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:43.536686 11271 consensus_queue.cc:1046] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20240507 08:05:43.561148 11271 raft_consensus.cc:3032] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.561638 11271 raft_consensus.cc:729] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.562012 11315 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.562577 11271 consensus_queue.cc:260] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.541296 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.566807 10304 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:43.567519 11358 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.568106 11358 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:43.568450 11358 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:43.568806 11358 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:43.542927 11401 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:43.546610 10938 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.536952 11464 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.548264 11268 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.549068 10191 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240507 08:05:43.549419 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 201474us
I20240507 08:05:43.549710 11258 consensus_queue.cc:1046] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:43.550074 11309 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.586308 10156 leader_election.cc:398] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4.
I20240507 08:05:43.586360 10906 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:43.550318 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50812.2us
I20240507 08:05:43.587478 10156 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.552713 11390 consensus_queue.cc:1046] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.570932 10337 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:43.550880 11409 consensus_queue.cc:1046] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.576825 11271 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.587428 11309 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240507 08:05:43.589072 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:43.597085 11470 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.597532 10906 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.597563 11470 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:43.598156 11268 raft_consensus.cc:3032] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.598691 11268 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.598839 10906 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.591403 10907 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.590857 10231 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.600288 11258 raft_consensus.cc:3032] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:43.600672 11258 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.601514 10231 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.601889 11390 consensus_queue.cc:1225] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240507 08:05:43.602304 11377 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.602409 11390 raft_consensus.cc:3032] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.602706 10231 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.602850 11390 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.602941 11377 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.603459 11319 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.605129 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.605167 11409 raft_consensus.cc:3032] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.605715 11409 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.606124 11319 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.606449 11409 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.607610 11409 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.599578 11447 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.608505 10231 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.609776 11467 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.609519 11258 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.609766 10304 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.610373 10906 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.610747 11258 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:43.611049 11255 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.611224 11447 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.611634 10906 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.603780 11390 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.612224 10304 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.612982 11390 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.613924 11467 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.615813 10935 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:43.600608 11268 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.625212 10156 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.622634 11447 raft_consensus.cc:2781] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:43.621246 11255 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.632064 10231 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.633512 10156 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.631950 10088 catalog_manager.cc:5484] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:43.636101 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:43.637568 10263 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:43.623903 11319 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.644029 11180 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.627056 11460 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.645172 11460 raft_consensus.cc:686] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.645924 11255 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:43.642982 11383 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:43.646373 11255 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:43.646831 11255 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.646770 11383 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.646551 11460 consensus_queue.cc:237] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.624011 11309 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20240507 08:05:43.598770 11470 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.653954 10191 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:43.655540 11180 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:43.621975 11358 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.662271 10088 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240507 08:05:43.672379 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08002fbba0 after lost signal to thread 9049
I20240507 08:05:43.672690 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:43.673372 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08002eaf40 after lost signal to thread 10124
W20240507 08:05:43.673730 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08002fe300 after lost signal to thread 10138
W20240507 08:05:43.674774 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2460 after lost signal to thread 10216
W20240507 08:05:43.675748 10362 debug-util.cc:398] Leaking SignalData structure 0x7b08002817a0 after lost signal to thread 10291
W20240507 08:05:43.676708 10362 debug-util.cc:398] Leaking SignalData structure 0x7b080021a900 after lost signal to thread 10365
W20240507 08:05:43.676959 10362 debug-util.cc:398] Leaking SignalData structure 0x7b080021a980 after lost signal to thread 10366
W20240507 08:05:43.679157 10362 debug-util.cc:398] Leaking SignalData structure 0x7b080032eda0 after lost signal to thread 10967
I20240507 08:05:43.668391 10088 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:43.687765 11383 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.687907 11447 raft_consensus.cc:686] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:43.677358 11268 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:43.689218 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 55814.2us
I20240507 08:05:43.689632 10088 catalog_manager.cc:5484] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:43.643234 11467 raft_consensus.cc:686] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:43.700376 10156 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.702519 10938 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20240507 08:05:43.703033 10938 pending_rounds.cc:76] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Aborting all ops after (but not including) 2
I20240507 08:05:43.703138 11358 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:43.703294 10938 pending_rounds.cc:97] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20240507 08:05:43.703653 10938 raft_consensus.cc:2894] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20240507 08:05:43.704116 10938 raft_consensus.cc:2866] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:43.705104 11464 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 LEARNER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240507 08:05:43.702447 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 74205us
I20240507 08:05:43.708647 11400 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.709224 11400 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.710570 11401 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:43.710214 11400 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:43.712774 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 84092.5us
I20240507 08:05:43.712679 11377 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:43.712707 11180 consensus_queue.cc:1046] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.714284 11377 raft_consensus.cc:3032] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.714630 11377 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.715473 11377 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.717856 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.732376 10156 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.732946 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.725311 11401 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.739905 11368 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.741305 11309 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.747066 10231 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.749327 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.750180 11255 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.752163 11255 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:43.753047 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:43.754557 11180 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.753937 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.756611 11336 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.754005 10090 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 2, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 2 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:43.759770 10259 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:43.761255 11401 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.764135 10231 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.746026 10156 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.773483 11467 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.745082 11377 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:43.791949 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 46029.9us
I20240507 08:05:43.793156 10156 leader_election.cc:398] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4.
I20240507 08:05:43.793542 10156 leader_election.cc:401] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240507 08:05:43.797089 10156 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:43.799126 11488 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240507 08:05:43.801376 10258 raft_consensus.cc:1261] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240507 08:05:43.805163 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 56005.5us
I20240507 08:05:43.807344 11478 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.807703 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.780028 11383 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.787266 11366 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.810997 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.791726 11271 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.812099 11460 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.795991 11401 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4.
I20240507 08:05:43.812271 11383 raft_consensus.cc:686] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:43.821306 11383 consensus_queue.cc:237] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.822082 10190 raft_consensus.cc:1226] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:43.745242 11447 consensus_queue.cc:237] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.814252 11366 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240507 08:05:43.829602 11491 log.cc:924] Time spent T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240507 08:05:43.831755 11258 consensus_queue.cc:1046] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:43.833125 11258 raft_consensus.cc:3032] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:43.835743 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.836627 11366 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.838307 10231 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.833527 11258 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:05:43.841405 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 44430.8us
I20240507 08:05:43.846251 10304 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.846797 11361 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.864056 10934 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.867835 10156 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.869333 11460 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:43.866513 11258 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.871920 11258 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:43.877173 10156 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.878106 11361 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.879004 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.881531 11358 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.881460 10304 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.882740 11358 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.883731 10263 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.884348 10304 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:43.887228 11361 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:43.888262 11361 raft_consensus.cc:686] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.889087 11361 consensus_queue.cc:237] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:43.900183 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 61997.7us
I20240507 08:05:43.901038 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.901180 11383 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.902835 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.900234 11366 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:43.918993 10231 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.918952 10156 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.923382 10266 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:43.923573 10267 raft_consensus.cc:1226] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:43.924774 10260 raft_consensus.cc:1226] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.925536 10264 raft_consensus.cc:1226] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.923223 10262 raft_consensus.cc:1226] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.927865 11366 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.928246 10231 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.928997 11366 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.929445 10231 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.927560 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:43.932232 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:43.934257 10338 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:43.935726 10156 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.935812 10304 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.936734 10304 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.939154 10332 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:43.942792 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:43.944068 11368 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.945269 10187 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:43.945401 11368 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.946770 11258 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.947809 11358 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.948071 10336 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:43.947674 10088 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 2 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:43.950502 11377 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:43.951107 11366 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:43.951714 10231 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:43.952551 11336 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.953279 11368 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.953598 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:43.953995 11258 consensus_queue.cc:1046] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.955137 11409 raft_consensus.cc:3032] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.955488 11409 raft_consensus.cc:729] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.956156 11366 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:43.956583 11366 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:43.956367 11409 consensus_queue.cc:260] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.958488 10187 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:43.961457 11358 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.961930 10156 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.956962 11366 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:43.964910 11258 consensus_queue.cc:1046] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.966185 11358 raft_consensus.cc:3032] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.966603 11358 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.966328 10088 catalog_manager.cc:5484] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 2 to 3, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 3 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:43.967567 11358 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.970427 11255 consensus_queue.cc:1046] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.973670 11428 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:43.976545 11255 raft_consensus.cc:3032] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.976909 11255 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:43.976739 10938 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:43.977802 11255 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:43.979112 11255 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.970954 11319 consensus_queue.cc:1046] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:43.981703 11401 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:43.986224 11358 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.986421 10191 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240507 08:05:43.987769 11409 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:43.989534 11377 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:43.995388 11309 raft_consensus.cc:3032] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:43.995843 11309 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:43.999518 10264 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:43.971450 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.972726 11180 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:43.974857 10156 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.003834 10336 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 3. (term mismatch)
I20240507 08:05:44.004186 10336 pending_rounds.cc:76] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 2
I20240507 08:05:44.004406 10336 pending_rounds.cc:97] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20240507 08:05:44.004650 10336 raft_consensus.cc:2894] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20240507 08:05:44.005003 10336 raft_consensus.cc:2866] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:44.006956 10156 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:43.997103 11309 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.011199 10938 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:44.011269 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:44.011906 10338 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.012213 11309 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.012603 11310 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.001260 11368 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.012848 11268 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240507 08:05:44.003562 11366 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.019424 10190 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240507 08:05:44.005991 11358 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.027539 11268 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240507 08:05:44.028321 10156 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.028420 10336 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:44.026480 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0f47233c3be4fe99e097ade2aaec1e8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:44.029438 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:43.976720 10187 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:44.031597 10156 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.031173 11368 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.042239 10156 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.025195 11400 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.023766 11401 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20240507 08:05:44.014878 11336 consensus_queue.cc:1159] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "59382e6647c549d89d83c70233c0667e" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20240507 08:05:44.014685 11377 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20240507 08:05:44.041328 11310 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:44.070423 10156 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.050145 11366 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.061468 11401 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:44.083601 11310 raft_consensus.cc:686] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:44.087054 11368 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.087527 10156 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.087288 11310 consensus_queue.cc:237] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.089075 11400 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.092131 10156 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.096704 10336 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:44.103924 10231 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:44.106173 10090 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:05:44.110941 10156 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.112514 10332 raft_consensus.cc:1226] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:44.112427 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:44.109294 10090 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:05:44.151225 10090 catalog_manager.cc:5484] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:05:44.154325 10262 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:44.157766 11368 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:44.157536 11467 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:44.158303 11368 raft_consensus.cc:686] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:05:44.158939 11507 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20240507 08:05:44.159291 11368 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:44.168452 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 68385.9us
I20240507 08:05:44.168844 11180 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:44.164502 11336 raft_consensus.cc:1050] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:05:44.170042 11336 raft_consensus.cc:1054] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Unable to promote non-voter 5d6656a616b747d393ff3de5328d2540: Illegal state: Leader has not yet committed an operation in its own term
I20240507 08:05:44.172114 11358 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:44.172657 11358 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.160588 11401 raft_consensus.cc:2741] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20240507 08:05:44.176693 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 80831.7us
I20240507 08:05:44.181062 10231 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:44.174933 11358 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.191089 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:44.218922 10935 raft_consensus.cc:2379] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:44.224936 10258 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.227438 10263 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240507 08:05:44.233847 11180 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:44.234272 11180 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:44.234639 11180 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.238389 11258 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:44.242139 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:44.247368 10264 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:44.250434 10304 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:44.234189 11467 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240507 08:05:44.245906 11377 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240507 08:05:44.271085 11180 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.253500 11358 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240507 08:05:44.284586 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 59774.1us
I20240507 08:05:44.310469 11180 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.280689 11319 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.245345 11510 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20240507 08:05:44.315748 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:44.319120 10190 raft_consensus.cc:2379] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:44.322418 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:44.285080 11358 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:44.326417 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 43615.8us
I20240507 08:05:44.327744 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:44.327982 11180 consensus_queue.cc:1046] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:44.334702 11180 raft_consensus.cc:3032] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:44.335067 11180 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:44.339152 11519 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.339651 11519 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.341643 11519 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.342604 10263 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.343039 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:44.331089 11358 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.354292 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:44.361227 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 208994us
I20240507 08:05:44.403281 11180 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:44.321027 11319 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.334100 11510 consensus_queue.cc:1046] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240507 08:05:44.408005 10258 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.404426 10156 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:44.352537 11511 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:44.417179 10155 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:44.414827 11512 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240507 08:05:44.427882 10263 raft_consensus.cc:3037] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:44.428222 11180 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:44.459136 11525 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20240507 08:05:44.460887 11520 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20240507 08:05:44.470557 10263 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240507 08:05:44.402827 11517 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20240507 08:05:44.412654 11478 consensus_queue.cc:1046] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20240507 08:05:44.495148 11478 raft_consensus.cc:3032] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:44.495522 11478 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:44.496342 11478 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.430068 10191 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.473448 11534 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.499202 11534 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.504402 11400 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240507 08:05:44.479266 11521 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.170s user 0.003s sys 0.000s
I20240507 08:05:44.475011 11460 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.487702 11368 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:44.492822 10183 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240507 08:05:44.493646 10339 raft_consensus.cc:1261] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20240507 08:05:44.473766 11524 log.cc:924] Time spent T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
W20240507 08:05:44.524890 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 120585us
I20240507 08:05:44.481124 11531 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240507 08:05:44.533727 10336 raft_consensus.cc:1261] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.539925 10336 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.547993 10337 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.550729 10341 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.556849 10340 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.558061 10332 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.561193 10338 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.563407 10335 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:44.518540 11534 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.566071 11467 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240507 08:05:44.567862 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.574195 11447 consensus_queue.cc:1035] T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:05:44.594048 10185 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.575182 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.595288 10232 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:44.576011 10333 raft_consensus.cc:1226] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:44.577523 11478 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.582105 11319 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.558966 10184 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:44.584158 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:44.599373 10938 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.584430 11541 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:44.596500 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 179483us
I20240507 08:05:44.547891 11309 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.600515 11541 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:44.607193 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 255287us
I20240507 08:05:44.606757 11361 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:44.598044 11534 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:44.616976 11541 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.613368 11368 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240507 08:05:44.621104 11271 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:44.625823 11368 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:44.625317 11271 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.627076 11383 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:44.628166 11271 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:44.629863 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.632997 10264 raft_consensus.cc:1226] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:44.610808 11527 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20240507 08:05:44.608886 11309 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.616219 11534 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:44.599100 11358 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.638506 11534 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:44.669607 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 318150us
I20240507 08:05:44.651051 11383 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20240507 08:05:44.677640 11390 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:44.678150 11390 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.679273 10334 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240507 08:05:44.678020 11529 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20240507 08:05:44.656190 11527 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240507 08:05:44.687472 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:44.688076 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.695214 11390 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.695631 11310 consensus_queue.cc:1046] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:44.696375 11358 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.696820 11400 raft_consensus.cc:3032] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:44.697172 11400 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:44.687145 10935 raft_consensus.cc:2932] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:44.678184 11255 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:44.697929 11400 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.698272 11255 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.700632 11255 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:44.701244 11511 raft_consensus.cc:3032] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:44.711194 11510 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.716683 11510 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.717630 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:44.717806 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:44.718413 10332 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.718762 11510 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:44.729772 11544 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20240507 08:05:44.676239 11517 consensus_queue.cc:1046] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20240507 08:05:44.713156 11511 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:44.739359 11511 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.739730 11447 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:44.740659 11511 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.684648 11356 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20240507 08:05:44.742789 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:44.745075 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.746086 10191 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:44.749218 10232 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:44.750173 11541 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:44.748651 10335 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.727003 11309 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.754712 11463 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.764125 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:44.764994 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:44.766870 11463 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.777704 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:44.778558 10259 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:44.780275 10231 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:44.759290 11534 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.750545 11541 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:44.793975 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:44.794620 10335 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.794899 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:44.771955 11390 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.783543 11538 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:44.786139 10262 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:44.787413 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.788019 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.790473 10231 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:44.802219 10190 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:44.803150 10231 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:44.795894 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:44.804282 10258 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:44.798329 11263 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:44.798538 11390 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.799068 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.799357 11258 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:44.768862 11463 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:44.809079 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:44.809314 11538 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.776477 10188 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:44.797307 11541 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:44.825096 11536 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:44.825618 11538 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.826361 11530 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.826891 11463 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.827009 11530 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.827420 11463 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.830245 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.830994 10186 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:44.831375 11463 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:44.833137 11390 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:44.834529 11309 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.834822 11534 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.835041 11309 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.812820 11336 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:44.837296 11336 raft_consensus.cc:686] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:44.838241 11336 consensus_queue.cc:237] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.839633 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:44.840391 10184 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:44.840535 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:44.841235 10935 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:44.842736 11447 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:44.819335 11258 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:44.844391 11258 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:44.846042 11537 raft_consensus.cc:3032] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:44.846413 11537 raft_consensus.cc:729] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:44.821746 10261 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:44.818323 10340 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.856391 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:44.858716 10186 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:44.859941 10232 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:44.821362 11409 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:44.860726 11409 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.861203 11319 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:44.861683 11319 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:44.862042 11319 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:44.862609 11409 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:44.863343 10935 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:44.837352 11309 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:44.876617 10233 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:44.879619 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 407587us
I20240507 08:05:44.879338 11309 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:44.885764 11309 raft_consensus.cc:686] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:44.886830 11309 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:44.875312 11530 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:44.873189 11537 consensus_queue.cc:260] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.899575 11537 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:44.902889 11541 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:44.936276 10086 catalog_manager.cc:5484] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:44.961483 10086 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:44.988246 10086 catalog_manager.cc:5484] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:44.958456 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:44.991200 10260 raft_consensus.cc:2354] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:05:45.018927 10086 catalog_manager.cc:5484] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:45.054234 11526 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.055097 11526 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.056728 11514 raft_consensus.cc:1050] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:05:45.057171 11526 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.060317 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.021392 10086 catalog_manager.cc:5484] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:05:45.062701 10086 catalog_manager.cc:5484] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20240507 08:05:45.053354 11525 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240507 08:05:45.060635 10934 raft_consensus.cc:2379] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:45.074729 11540 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.075976 11526 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.066881 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:45.103860 10935 raft_consensus.cc:2932] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:45.106402 10258 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42910
I20240507 08:05:45.133566 10938 raft_consensus.cc:2379] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:45.112186 11526 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:45.145980 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 473582us
I20240507 08:05:45.102231 11540 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:45.152473 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 478160us
I20240507 08:05:45.145151 11319 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.172505 11351 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:45.207720 11514 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:45.196698 10187 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.020s sys 0.025s
I20240507 08:05:45.163956 11258 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:45.232852 10333 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.037s sys 0.094s
I20240507 08:05:45.202004 11401 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:45.241472 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 445813us
I20240507 08:05:45.250124 10231 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:45.248781 11401 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.251349 11536 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:45.251719 11536 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:45.252070 11536 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:45.252249 11401 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:45.241174 10187 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.021s sys 0.025s
I20240507 08:05:45.204998 10184 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
W20240507 08:05:45.247714 10333 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s user 0.037s sys 0.094s
I20240507 08:05:45.247215 11463 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.247463 11530 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.263212 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9adf1535c974b2abf910f3c1c537627 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:45.269120 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:45.270038 10261 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:45.248716 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:45.274470 10258 raft_consensus.cc:595] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Received request to transfer leadership
I20240507 08:05:45.281833 11535 raft_consensus.cc:1013] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: attempt to promote peer 5d6656a616b747d393ff3de5328d2540: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240507 08:05:45.286347 11540 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.286443 11268 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.287099 11268 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.287335 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.288213 10186 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:45.289099 11268 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.289824 10908 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:45.290431 11526 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.292069 11540 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:45.292436 11540 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:45.292744 11540 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:45.292805 11258 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:45.294750 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:45.300623 10262 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:45.303788 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:45.305799 11319 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.306674 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:45.273571 11463 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.271905 11530 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:45.326653 10306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.004s sys 0.031s
W20240507 08:05:45.329268 10306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.004s sys 0.031s
W20240507 08:05:45.329614 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 532213us
I20240507 08:05:45.327512 10934 raft_consensus.cc:2379] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 2: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
W20240507 08:05:45.331302 10334 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.012s sys 0.034s
I20240507 08:05:45.394317 11358 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.394934 11263 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.396800 11358 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.397397 11263 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.399422 11463 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.401073 11530 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.401912 11263 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:45.402746 11537 consensus_meta.cc:287] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.503s user 0.009s sys 0.003s
I20240507 08:05:45.405292 10306 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:45.406649 11463 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240507 08:05:45.385730 10334 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s user 0.013s sys 0.034s
W20240507 08:05:45.411496 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 210244us
I20240507 08:05:45.396670 10939 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240507 08:05:45.391997 11271 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:45.413792 11536 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.426214 11531 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:45.413508 11558 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:45.427424 11553 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20240507 08:05:45.427935 10261 raft_consensus.cc:1261] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:45.434660 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:45.435637 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:45.435945 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:45.465281 10304 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:45.465471 10332 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:45.430965 11551 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.395138 11336 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.461786 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.462596 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.470605 10186 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:45.432413 11268 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.466981 10267 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:45.467478 10304 leader_election.cc:398] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:05:45.481783 10304 leader_election.cc:401] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:05:45.463526 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.431914 11271 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.461218 11309 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:45.439635 11263 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20240507 08:05:45.505504 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.506354 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:45.434382 11531 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:45.479635 11358 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.469878 11361 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.480149 11530 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:45.535032 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:45.535707 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.482554 11463 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:45.542050 10332 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:45.493012 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.481940 11366 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20240507 08:05:45.467325 11558 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.479895 11551 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.727137 10306 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:45.661077 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 178249us
I20240507 08:05:45.582265 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:45.735550 10183 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:45.582094 11428 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:45.738277 11571 log.cc:924] Time spent T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20240507 08:05:45.533133 11467 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.708151 11460 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:45.530853 11400 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:45.581171 10191 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
W20240507 08:05:45.703166 11560 log.cc:924] Time spent T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20240507 08:05:45.532184 11559 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20240507 08:05:45.710723 11550 log.cc:924] Time spent T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
I20240507 08:05:45.773662 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:45.591601 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:45.621469 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:45.547094 11547 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240507 08:05:45.532707 11536 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.531787 11541 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:45.635937 11549 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.069s user 0.000s sys 0.001s
W20240507 08:05:45.532902 11563 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:05:45.530288 11271 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.808718 11580 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:45.809147 11580 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:45.705459 11581 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.765975 11531 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:45.780550 11569 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20240507 08:05:45.781718 11570 log.cc:924] Time spent T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20240507 08:05:45.812914 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:45.499063 11268 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:45.653409 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:45.607479 11358 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 4
I20240507 08:05:45.779054 10186 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:45.703639 11258 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:45.528844 11555 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:45.547293 11514 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.733970 11463 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:45.766963 11554 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.289s user 0.000s sys 0.001s
I20240507 08:05:45.680413 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:45.864470 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.865736 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.865770 10336 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:45.866724 11268 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:45.866894 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.764338 11319 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.869830 11319 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.871903 11319 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.703454 10939 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240507 08:05:45.590817 11530 raft_consensus.cc:686] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:45.702358 11542 consensus_queue.cc:1035] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.306s
I20240507 08:05:45.816954 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:45.547398 11361 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.876309 11361 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.774109 10187 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:45.501041 11336 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.877272 11530 consensus_queue.cc:237] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:45.879433 11336 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.809520 11580 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:45.881008 11588 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:45.881556 11588 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:45.856143 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 235152us
I20240507 08:05:45.883489 11588 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.884276 11573 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.884769 11573 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.886847 11573 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.889874 11580 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.892352 11580 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.764400 11309 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.897403 11309 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.899647 11263 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:45.900065 11263 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:45.900403 11263 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:45.903239 11351 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:45.903734 11351 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:45.905668 11514 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.907753 11514 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.909885 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 830) took 1208 ms. Trace:
I20240507 08:05:45.910283 10338 rpcz_store.cc:276] 0507 08:05:44.701875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:44.704026 (+ 2151us) service_pool.cc:225] Handling call
0507 08:05:44.717310 (+ 13284us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:05:44.752322 (+ 35012us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:44.752366 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:05:44.788978 (+ 36612us) log.cc:842] Serialized 52 byte log entry
0507 08:05:45.188247 (+399269us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:45.463867 (+275620us) log.cc:842] Serialized 14 byte log entry
0507 08:05:45.747235 (+283368us) log.cc:842] Serialized 14 byte log entry
0507 08:05:45.774861 (+ 27626us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:45.774925 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:45.909633 (+134708us) raft_consensus.cc:1671] finished
0507 08:05:45.909655 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:45.909806 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90368,"prepare.queue_time_us":488005,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":68,"raft.queue_time_us":203353,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":70573,"thread_start_us":798370,"threads_started":3,"wal-append.queue_time_us":542832}
I20240507 08:05:45.861423 11463 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:45.815201 11581 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.929724 11581 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.812474 11579 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:45.930508 11579 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:45.932562 11579 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.954883 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 42) took 1277 ms. Trace:
I20240507 08:05:45.955391 10333 rpcz_store.cc:276] 0507 08:05:44.677849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:44.678252 (+ 403us) service_pool.cc:225] Handling call
0507 08:05:44.678573 (+ 321us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:44.678895 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:44.678926 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:44.679387 (+ 461us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.297104 (+617717us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:05:45.300598 (+ 3494us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:45.302103 (+ 1505us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:45.954661 (+652558us) raft_consensus.cc:1671] finished
0507 08:05:45.954683 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:45.954823 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1012,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":7124,"thread_start_us":580641,"threads_started":1,"wal-append.queue_time_us":700083}
I20240507 08:05:45.738580 11428 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.964151 11428 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.770255 11460 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.967497 11460 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.862391 11258 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:45.978840 11258 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:45.825593 11531 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.980203 11531 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:45.982412 11531 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.770432 11467 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:45.989513 11467 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:45.992435 10337 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:45.994329 11511 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:45.994838 11511 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:45.996742 11511 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:45.997776 11554 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240507 08:05:46.006717 11383 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:46.007351 11383 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.007995 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.008636 10337 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:46.010959 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 850) took 1365 ms. Trace:
I20240507 08:05:46.011320 10263 rpcz_store.cc:276] 0507 08:05:44.645795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:44.645900 (+ 105us) service_pool.cc:225] Handling call
0507 08:05:44.646172 (+ 272us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:44.648190 (+ 2018us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:44.648226 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:44.651745 (+ 3519us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.166305 (+514560us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:45.166355 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:45.166448 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:45.535296 (+368848us) spinlock_profiling.cc:243] Waited 9.95 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:46.010661 (+475365us) raft_consensus.cc:1671] finished
0507 08:05:46.010684 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.010883 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13421,"raft.queue_time_us":3786,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":3211,"spinlock_wait_cycles":22883584,"thread_start_us":497512,"threads_started":1,"wal-append.queue_time_us":754818}
I20240507 08:05:46.012423 10907 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.022410 10333 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20240507 08:05:46.033210 11540 consensus_meta.cc:287] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.740s user 0.009s sys 0.002s
I20240507 08:05:46.033692 11540 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:46.036530 11555 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:46.039672 11555 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:46.040333 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.041079 10189 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:46.009296 11383 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.043414 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.044883 10338 raft_consensus.cc:2379] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:46.043601 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.045850 10192 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:46.046321 10336 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:46.048218 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.050246 10183 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
W20240507 08:05:46.054428 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 290548us
I20240507 08:05:45.997228 11511 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20240507 08:05:46.063207 10306 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.064319 11409 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.064710 11409 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.065028 11409 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:46.066268 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 291912us
I20240507 08:05:46.051204 11526 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:46.056738 11559 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.203s user 0.000s sys 0.000s
I20240507 08:05:46.076213 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:46.077329 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:46.087610 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 40) took 1241 ms. Trace:
I20240507 08:05:46.088845 11581 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.089320 11581 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.089701 11581 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.094198 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:46.095088 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.095566 10189 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240507 08:05:46.089337 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:46.097360 10187 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:46.097255 10907 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:46.098129 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.099443 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.099738 10183 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240507 08:05:46.095927 10263 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:46.100371 10192 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:46.103125 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:46.106284 11586 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:46.110831 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.123395 10189 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:46.090435 11583 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.100087 10306 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.088032 10185 rpcz_store.cc:276] 0507 08:05:44.845908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:44.846069 (+ 161us) service_pool.cc:225] Handling call
0507 08:05:44.846471 (+ 402us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:44.846819 (+ 348us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:44.846854 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:44.847321 (+ 467us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.257125 (+409804us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:05:45.277623 (+ 20498us) log.cc:842] Serialized 14 byte log entry
0507 08:05:45.298898 (+ 21275us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:45.300820 (+ 1922us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:46.087394 (+786574us) raft_consensus.cc:1671] finished
0507 08:05:46.087424 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.087554 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17422,"prepare.queue_time_us":4242,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":22260,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":10538,"thread_start_us":405403,"threads_started":1,"wal-append.queue_time_us":537988}
I20240507 08:05:46.133888 11263 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.126649 11583 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.091214 11526 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.153913 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.152740 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.156176 10186 raft_consensus.cc:2379] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:46.112030 11377 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.155637 10189 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:46.157225 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.152673 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.152330 11511 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.160336 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.160264 11511 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.160972 10267 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:46.149183 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:46.159765 10260 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:46.134397 11428 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.169319 10231 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:46.180269 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.181210 10260 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:46.104380 11540 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:46.164746 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 64766.4us
I20240507 08:05:46.180269 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.185433 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.186024 10333 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:46.194885 10232 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:46.153276 11510 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.189406 11511 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.189620 11319 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.158022 11583 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.191360 11263 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.184334 11558 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.194036 10189 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:46.194064 10261 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:46.194700 10306 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.188323 11586 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:46.156864 11535 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.197497 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:46.229868 10262 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.199689 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.200816 10260 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240507 08:05:46.217307 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:46.179306 11534 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:46.238552 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.179359 11258 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:46.220258 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.240347 10259 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:46.240648 10232 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.192790 11551 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.236724 10306 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.177305 10263 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:46.223227 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.238710 11534 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.202417 11510 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.166026 11377 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:46.237865 11366 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:46.237159 11527 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.272719 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.243611 11358 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:05:46.243988 11558 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.249792 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:46.249809 11336 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:46.250469 11511 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:46.250514 11263 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:46.245308 11478 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.245589 11258 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:46.288556 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 470239us
I20240507 08:05:46.205871 11319 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.252522 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.253235 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.290205 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.290343 10338 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:46.290823 10263 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:46.290813 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.254308 10336 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:46.291570 10261 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:46.248737 10262 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
W20240507 08:05:46.261427 10341 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.027s sys 0.029s
W20240507 08:05:46.262367 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 46716.5us
W20240507 08:05:46.292420 10341 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.789s user 0.028s sys 0.029s
I20240507 08:05:46.223227 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.243268 11526 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.120415 11463 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:46.254691 10340 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.619s user 0.017s sys 0.041s
I20240507 08:05:46.196924 11428 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.275704 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.280057 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:46.303128 10186 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.267774 11534 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:46.286119 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.251472 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:46.293035 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 859) took 1114 ms. Trace:
I20240507 08:05:46.297297 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:46.298907 10339 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.000s sys 0.033s
I20240507 08:05:46.299784 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.300370 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:46.300539 10156 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.005s sys 0.009s
I20240507 08:05:46.297492 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.304500 10189 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
W20240507 08:05:46.314566 10156 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.006s sys 0.009s
W20240507 08:05:46.315186 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 672401us
I20240507 08:05:46.317252 10906 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:46.277264 11558 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.300139 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 857) took 1122 ms. Trace:
I20240507 08:05:46.321568 10335 rpcz_store.cc:276] 0507 08:05:45.177944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:45.178430 (+ 486us) service_pool.cc:225] Handling call
0507 08:05:45.201479 (+ 23049us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:45.215511 (+ 14032us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:45.215541 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:45.230896 (+ 15355us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.466952 (+236056us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:45.807510 (+340558us) log.cc:842] Serialized 14 byte log entry
0507 08:05:45.926613 (+119103us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b24002916c8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dd58024 00007fcf8dd5613b 00007fcf8dde4149 00007fcf8dde3dd5 00007fcf8dde3330 00007fcf90ad63a8 00007fcf90ad6b08 00007fcf90aae618 00007fcf90aa9a1a 00007fcf90aa8e66
0507 08:05:46.275938 (+349325us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:46.299713 (+ 23775us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:46.299787 (+ 74us) raft_consensus.cc:1671] finished
0507 08:05:46.299896 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.300065 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28221,"prepare.queue_time_us":237082,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":24,"raft.queue_time_us":117213,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":3821,"spinlock_wait_cycles":158955392,"thread_start_us":947264,"threads_started":3,"wal-append.queue_time_us":386099}
I20240507 08:05:46.294847 11263 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.311866 10191 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:46.337158 10264 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:46.333106 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:46.310168 10341 rpcz_store.cc:276] 0507 08:05:45.178590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:45.178757 (+ 167us) service_pool.cc:225] Handling call
0507 08:05:45.179132 (+ 375us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:45.179546 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:45.179587 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:45.188504 (+ 8917us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.412748 (+224244us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:45.502516 (+ 89768us) log.cc:842] Serialized 14 byte log entry
0507 08:05:46.292749 (+790233us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:46.292818 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:46.292860 (+ 42us) raft_consensus.cc:1671] finished
0507 08:05:46.292872 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.292989 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19799,"prepare.queue_time_us":283163,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":177976,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":31332,"spinlock_wait_cycles":1738496,"thread_start_us":1049701,"threads_started":3,"wal-append.queue_time_us":221536}
W20240507 08:05:46.302704 10340 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.671s user 0.018s sys 0.041s
I20240507 08:05:46.295936 11319 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.294569 11258 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:46.239991 11535 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.273056 11351 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.299418 11511 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:46.359634 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.332798 11558 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.348476 11263 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:46.316519 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:46.331275 10334 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.295432 11401 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.319597 11573 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
W20240507 08:05:46.319331 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 220613us
I20240507 08:05:46.368149 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:46.368671 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52351us
W20240507 08:05:46.309700 10155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.007s sys 0.004s
I20240507 08:05:46.330904 11540 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.370404 10341 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:46.399232 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.399874 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.400542 10341 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:46.401650 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:46.402719 10155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.007s sys 0.007s
I20240507 08:05:46.406008 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.406137 11540 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.410243 10266 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:46.410516 11540 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.412631 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.413245 10338 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:46.374447 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:46.415535 11591 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.415585 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:46.310671 10339 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.667s user 0.000s sys 0.034s
W20240507 08:05:46.371142 10332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.013s sys 0.012s
W20240507 08:05:46.419343 10332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.013s sys 0.012s
I20240507 08:05:46.304991 10185 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:46.337193 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.364276 10186 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:46.297029 11478 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.277558 11527 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.329128 10337 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:46.390225 10335 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:46.392643 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:46.383606 11593 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.421191 10156 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:46.427037 10262 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:05:46.417461 11336 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.440680 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:46.438498 10263 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:46.430420 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.443698 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.418352 11556 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:46.448405 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:46.448663 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 858) took 1270 ms. Trace:
I20240507 08:05:46.449198 10335 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:46.449144 10339 rpcz_store.cc:276] 0507 08:05:45.178279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:45.179663 (+ 1384us) service_pool.cc:225] Handling call
0507 08:05:45.203216 (+ 23553us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:45.245026 (+ 41810us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:45.245070 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:45.245566 (+ 496us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.517536 (+271970us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:46.427462 (+909926us) log.cc:842] Serialized 14 byte log entry
0507 08:05:46.448247 (+ 20785us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:46.448341 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:46.448400 (+ 59us) raft_consensus.cc:1671] finished
0507 08:05:46.448417 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.448602 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164841,"prepare.queue_time_us":186789,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":42498,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5651,"thread_start_us":1041981,"threads_started":2,"wal-append.queue_time_us":452196}
I20240507 08:05:46.444489 10261 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:46.455199 10190 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:46.454416 10304 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:46.392555 11401 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:46.459136 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 217324us
I20240507 08:05:46.461885 10259 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:46.437400 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 856) took 1263 ms. Trace:
I20240507 08:05:46.441043 11336 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.463764 11510 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.464018 11336 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.464778 11409 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.466786 11409 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.447829 10938 raft_consensus.cc:2379] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:46.467053 11583 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:46.469699 10140 thread.cc:636] client-negotiator [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.005s sys 0.032s
W20240507 08:05:46.470027 10140 thread.cc:602] client-negotiator [worker] (thread pool) Time spent starting thread: real 0.964s user 0.006s sys 0.032s
I20240507 08:05:46.469985 11535 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.435616 11511 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.471030 11579 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:46.471529 11579 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:05:46.463662 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:46.475586 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 54509.3us
I20240507 08:05:46.476181 10261 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:46.478112 10333 raft_consensus.cc:2379] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:46.463472 10340 rpcz_store.cc:276] 0507 08:05:45.171246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:45.179581 (+ 8335us) service_pool.cc:225] Handling call
0507 08:05:45.207755 (+ 28174us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:45.245918 (+ 38163us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:45.249112 (+ 3194us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:45.282766 (+ 33654us) log.cc:842] Serialized 26 byte log entry
0507 08:05:45.478233 (+195467us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:45.882065 (+403832us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b4800180258. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:05:46.370161 (+488096us) log.cc:842] Serialized 14 byte log entry
0507 08:05:46.417971 (+ 47810us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:46.418064 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:46.418113 (+ 49us) raft_consensus.cc:1671] finished
0507 08:05:46.421410 (+ 3297us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.434817 (+ 13407us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155639,"prepare.queue_time_us":225910,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":98118,"raft.run_cpu_time_us":8448,"raft.run_wall_time_us":643849,"spinlock_wait_cycles":5124352,"thread_start_us":910820,"threads_started":2,"wal-append.queue_time_us":527691}
I20240507 08:05:46.443676 11593 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.473491 11579 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.419059 11538 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240507 08:05:46.484319 10338 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:46.485889 10262 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:46.488054 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.491079 10258 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:46.491729 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.494132 10262 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
W20240507 08:05:46.440925 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 255701us
I20240507 08:05:46.499612 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.416138 11591 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:46.505129 10261 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:46.506951 10232 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.460608 11580 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.489302 11517 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:46.504220 11533 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.596904 11533 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.498924 11535 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.497644 11463 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.590340 11514 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.497102 11526 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.101s
I20240507 08:05:46.495222 11573 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20240507 08:05:46.593530 11527 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.593323 11428 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.609081 11535 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.654223 11535 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.610050 11533 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.594316 11517 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:46.650871 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 196615us
I20240507 08:05:46.589795 11542 consensus_queue.cc:1046] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20240507 08:05:46.666316 10335 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:05:46.592748 11580 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.620582 11514 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.675547 10304 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.551975 11579 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20240507 08:05:46.691231 10304 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:46.635186 11539 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.002s sys 0.007s
W20240507 08:05:46.660892 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 221585us
I20240507 08:05:46.696028 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
W20240507 08:05:46.686482 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 182261us
I20240507 08:05:46.660984 11583 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.719763 10304 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.677675 11517 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:46.720810 11361 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.734831 11361 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.735195 11361 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.739004 11609 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.739547 11609 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.742131 11609 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:46.712730 11539 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.591s user 0.003s sys 0.007s
I20240507 08:05:46.690378 11514 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.754094 10306 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:46.754282 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 258210us
I20240507 08:05:46.755497 10306 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:46.748395 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:46.699440 11592 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.717182 11530 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:46.761071 11358 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:46.760196 10304 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.728055 11564 raft_consensus.cc:3032] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:05:46.689854 11534 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.765209 11598 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:46.764506 11271 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:46.725057 11529 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:46.777983 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 253729us
I20240507 08:05:46.691109 11258 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.792614 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.793421 10339 raft_consensus.cc:2379] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:46.782048 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.782903 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.744979 11531 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.794770 10341 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:46.786012 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:46.787747 11358 raft_consensus.cc:686] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:46.717029 11590 consensus_queue.cc:1046] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20240507 08:05:46.759941 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.766084 11609 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.757047 11478 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.758464 11593 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:46.768855 11562 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.031s sys 0.023s
I20240507 08:05:46.797650 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.799661 11614 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:46.799417 11358 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:46.799980 11268 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.792680 11519 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.843674 11519 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.800705 11377 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:46.846160 11519 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.769135 11538 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.804206 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.850098 10335 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:46.757503 11336 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.779857 11564 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:46.788002 11530 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
W20240507 08:05:46.836175 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44179.5us
I20240507 08:05:46.834389 11594 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:46.790098 10907 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:46.808552 11361 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.843132 11268 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:46.851613 11610 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20240507 08:05:46.887679 11628 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.888247 11628 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.888620 11628 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.810945 11609 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.852221 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.811940 11263 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:46.841470 11562 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s user 0.032s sys 0.024s
I20240507 08:05:46.802984 11598 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:46.859160 10232 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.803834 11529 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:46.863318 11401 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.863942 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 865) took 1678 ms. Trace:
I20240507 08:05:46.867470 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.921897 10264 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:46.803273 11531 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:46.873029 11319 consensus_meta.cc:287] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.504s user 0.009s sys 0.000s
I20240507 08:05:46.925302 11319 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.873757 11530 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:46.840049 11542 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:46.837172 11591 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.771250 11592 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.877650 10337 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:46.882329 11268 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:46.943178 10334 consensus_meta.cc:287] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.536s user 0.008s sys 0.000s
I20240507 08:05:46.943552 10334 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:46.893240 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.945281 10335 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:46.885294 11534 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:46.905244 11611 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.192s user 0.001s sys 0.000s
I20240507 08:05:46.910322 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:46.912155 10262 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:46.914594 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.879177 11580 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.915961 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:46.916239 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 57174.9us
I20240507 08:05:46.918941 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:46.924268 11625 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:46.924492 11361 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:46.987617 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.920166 10332 rpcz_store.cc:276] 0507 08:05:45.185842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:45.775921 (+590079us) service_pool.cc:225] Handling call
0507 08:05:45.776262 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:46.419696 (+643434us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:46.419729 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:46.439935 (+ 20206us) log.cc:842] Serialized 26 byte log entry
0507 08:05:46.706077 (+266142us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:46.706130 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:46.706218 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:46.863712 (+157494us) raft_consensus.cc:1671] finished
0507 08:05:46.863735 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:46.863880 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":125834,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":861603,"threads_started":2,"wal-append.queue_time_us":331004}
I20240507 08:05:46.933302 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:46.934522 10336 raft_consensus.cc:2354] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
I20240507 08:05:46.856810 11538 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.913970 11598 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:46.934098 11319 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.941241 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.000995 10337 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
W20240507 08:05:46.909265 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 130913us
I20240507 08:05:46.977756 10261 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:46.934697 11428 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:46.978032 10259 raft_consensus.cc:2379] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:46.978260 10341 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:46.982695 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:46.985411 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:47.009647 10262 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:46.987053 11511 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:46.903470 11540 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:46.938228 11401 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20240507 08:05:47.000044 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:46.990180 10187 consensus_meta.cc:287] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.892s user 0.009s sys 0.003s
I20240507 08:05:46.950343 11609 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:46.967600 11551 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.944600 11529 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:47.015796 11535 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.001031 11526 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:47.022722 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 89537.6us
I20240507 08:05:47.027227 10189 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:46.999981 11527 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:46.939674 11564 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:46.939512 11542 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.036659 11478 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:47.058012 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:47.059351 10262 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:47.039252 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:47.014416 11428 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:47.028118 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.020977 11625 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:46.999975 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:47.028291 11511 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.074565 11400 consensus_queue.cc:1046] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:47.024832 11258 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.036526 11309 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:47.081686 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.082552 10155 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:47.055454 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:47.036706 10187 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:05:47.044679 11530 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.047755 11551 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.101744 10156 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:47.105479 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.062415 11527 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.133124 10335 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:05:47.040059 11592 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.112321 10190 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:47.077739 11428 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:47.066009 11478 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:47.141428 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 59190.2us
I20240507 08:05:47.077725 10938 raft_consensus.cc:1226] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:47.078393 11336 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.087808 11564 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:47.049006 11526 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.095443 11627 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:47.100795 11535 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.102370 11538 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.113539 10337 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
W20240507 08:05:47.135511 11581 consensus_meta.cc:287] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 1.045s user 0.012s sys 0.001s
I20240507 08:05:47.122151 11511 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.131554 11263 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.136937 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 44) took 1047 ms. Trace:
I20240507 08:05:47.134039 10185 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:47.151024 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:47.099501 11628 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.145246 11478 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:47.099995 11268 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.123819 11309 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:47.155695 11558 consensus_meta.cc:287] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.777s user 0.003s sys 0.010s
W20240507 08:05:47.192229 11533 consensus_meta.cc:287] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.535s user 0.011s sys 0.000s
W20240507 08:05:47.202207 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 442542us
W20240507 08:05:47.202888 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 426446us
I20240507 08:05:47.246840 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.247500 10335 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:47.248407 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.249902 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.249348 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.250689 10332 raft_consensus.cc:2379] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:47.251003 10339 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:47.220816 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:47.224910 10341 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240507 08:05:47.101209 11625 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:47.184338 11460 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:47.231741 11621 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:47.232419 10336 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:05:47.186443 11627 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:47.101419 11619 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.127s
I20240507 08:05:47.246989 11556 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:47.216257 11605 log.cc:924] Time spent T 50d914d033cf48c389262dd8b7c7fbce P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.483s user 0.005s sys 0.002s
I20240507 08:05:47.257247 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.243069 11540 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.280467 10336 raft_consensus.cc:595] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:05:47.283376 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:47.233850 10184 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:47.199908 10187 rpcz_store.cc:276] 0507 08:05:46.087212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.089013 (+ 1801us) service_pool.cc:225] Handling call
0507 08:05:47.134990 (+1045977us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:47.190338 11581 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:47.249326 11263 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:47.278350 11460 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:47.331895 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.302817 11627 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:47.308135 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:47.337953 10332 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:47.311034 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:47.339170 10191 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:47.251713 11258 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:47.253460 11592 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:47.228272 11622 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:47.744318 11534 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:47.744086 11622 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.744503 10232 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:47.744848 11534 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:47.307220 11548 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.028s sys 0.017s
I20240507 08:05:47.242287 11558 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:47.745939 11548 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.156s user 0.029s sys 0.017s
I20240507 08:05:47.747890 11548 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:47.748169 11558 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.368932 11258 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.748555 11548 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.419695 11527 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.474439 11581 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:47.420452 11606 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.021s sys 0.039s
W20240507 08:05:47.750922 11606 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.135s user 0.024s sys 0.039s
I20240507 08:05:47.750979 11622 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:47.473781 11631 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20240507 08:05:47.369971 11551 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:47.344362 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 242817us
W20240507 08:05:47.396051 11624 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.330s user 0.000s sys 0.002s
I20240507 08:05:47.340181 11263 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.753098 10156 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:47.753933 11628 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.754133 11263 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.472373 10191 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:47.754917 10156 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:47.755620 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:47.249974 11514 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:47.756331 11598 consensus_meta.cc:287] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.741s user 0.009s sys 0.003s
I20240507 08:05:47.756731 11598 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.757414 11551 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:47.758105 11514 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.323997 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:47.758955 10185 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:05:47.756752 10156 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:47.329375 11542 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.310827 11556 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.250288 11533 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:47.406280 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 420758us
I20240507 08:05:47.345309 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:47.455426 11351 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.412220 11460 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.457087 11526 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.458232 10332 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:47.767304 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:47.768582 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:47.516319 11268 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.774971 11534 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:47.775878 11478 consensus_meta.cc:287] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.001s
I20240507 08:05:47.776381 11478 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
W20240507 08:05:47.777782 11517 consensus_meta.cc:287] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 1.022s user 0.012s sys 0.008s
I20240507 08:05:47.780912 11517 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.788271 11409 raft_consensus.cc:3032] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:47.788722 11409 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:47.807090 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:47.468199 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 44223.4us
W20240507 08:05:47.605335 11644 log.cc:924] Time spent T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240507 08:05:47.312885 11579 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:47.727872 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:47.729230 11601 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:05:47.542093 10192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.011s sys 0.009s
W20240507 08:05:47.926138 10192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s user 0.013s sys 0.032s
I20240507 08:05:47.730000 11309 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:47.734179 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:47.317663 11336 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:47.742605 11638 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:47.749243 10337 raft_consensus.cc:2354] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:47.758010 11551 consensus_queue.cc:1046] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:47.423022 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:47.325237 11401 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:47.756350 10191 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:47.761446 11556 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.764565 11467 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:47.764801 10340 raft_consensus.cc:1226] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:47.764953 11533 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.765462 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:47.958304 10336 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:47.898972 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 138264us
I20240507 08:05:47.959514 10233 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:47.961890 10906 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:47.919991 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:47.926992 10188 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:47.789543 11409 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.854357 11541 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:05:47.931746 10908 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.764s (SENT) [suppressed 51 similar messages]
I20240507 08:05:47.949841 10334 raft_consensus.cc:2379] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:47.950763 11478 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:47.951505 11517 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:47.952558 11428 consensus_meta.cc:287] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.804s user 0.006s sys 0.007s
I20240507 08:05:47.952821 11401 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:47.953495 11581 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:47.955293 11548 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:47.965535 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:47.972841 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:47.993678 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:48.044742 10259 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:47.987103 10191 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:48.009423 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:48.046414 10263 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:48.010740 10332 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:48.047979 10332 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:47.744953 11592 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.946034 11638 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:47.966466 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:47.932480 10934 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:47.945747 11309 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:48.028105 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:47.946650 11534 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.006814 11377 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:48.007686 11541 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:48.054086 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.040943 11478 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:48.090761 10263 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:48.054992 10188 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:48.056943 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.058995 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:48.049078 10336 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:47.941495 11601 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:47.971689 11556 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.106318 11556 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.079842 11606 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.107676 11606 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.085068 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:48.109588 10261 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 5
W20240507 08:05:48.110016 11627 consensus_meta.cc:287] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.714s user 0.005s sys 0.007s
I20240507 08:05:48.090097 11534 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.052522 11542 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:48.041429 11401 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:47.947256 11336 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.092059 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:48.091722 10191 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:48.102062 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.103499 11621 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:48.105401 11601 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:48.093632 11541 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:48.106771 11598 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:48.069761 10258 raft_consensus.cc:2379] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:47.971408 11467 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.116963 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.090498 11558 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:48.078554 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.036919 11428 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:48.094461 11478 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.122944 10261 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:48.053112 11581 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.110404 11627 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.121366 10259 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:47.926678 11579 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.124331 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.128221 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:48.139838 10191 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:48.133001 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:48.141966 10263 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:48.141953 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:48.142802 10259 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:48.120958 11621 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:48.118119 11642 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:48.125841 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 381573us
I20240507 08:05:48.133899 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:48.123566 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:48.144888 10260 raft_consensus.cc:2379] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:48.131593 11517 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:48.147315 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:48.150347 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 252072us
I20240507 08:05:48.152700 10306 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:48.165282 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:48.165904 10306 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:48.132926 11478 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:48.154731 10261 raft_consensus.cc:2354] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:48.161366 10232 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:48.133616 10265 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:48.179426 10232 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:48.173257 10156 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:48.130692 11558 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.174409 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:48.174188 10259 raft_consensus.cc:2454] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:48.187119 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.088902 11309 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.174468 11598 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:48.162933 11601 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.179334 11617 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:48.183029 11351 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:48.175845 11600 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:48.187971 11649 log.cc:924] Time spent T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20240507 08:05:48.204542 10232 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:48.190563 11628 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:48.187002 10185 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:48.169746 11517 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.154559 11642 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.206284 11598 raft_consensus.cc:686] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:48.218685 10232 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:48.201311 11351 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:48.203802 11650 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:48.211318 11656 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:48.214854 11606 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.216073 11617 raft_consensus.cc:686] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:05:48.212142 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 852) took 1957 ms (client timeout 2031 ms). Trace:
I20240507 08:05:48.216769 11646 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.199107 11601 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.222780 11592 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:48.249670 11556 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:48.262320 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 204534us
I20240507 08:05:48.251319 11658 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.242153 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 899) took 1109 ms. Trace:
W20240507 08:05:48.278740 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 105866us
I20240507 08:05:48.250450 11351 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:48.271554 11601 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:48.217904 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 292291us
I20240507 08:05:48.261925 11654 raft_consensus.cc:2741] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240507 08:05:48.270593 11579 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.271041 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:48.206350 11600 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.276119 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:48.242781 11534 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:48.270651 11647 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
I20240507 08:05:48.246311 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:48.269857 11646 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.274150 11606 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:48.278712 11467 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:48.273655 11627 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.272619 11428 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:48.423151 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 147223us
I20240507 08:05:48.424710 10231 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:48.307200 11645 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20240507 08:05:48.287955 11556 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.285305 11658 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.306929 11638 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.286652 11581 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:48.391237 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:48.439786 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:48.405439 11519 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.333076 11617 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.272089 11656 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.395826 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:48.254438 11650 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.288054 11377 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:48.452045 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 888) took 1465 ms. Trace:
I20240507 08:05:48.431754 11627 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:48.452529 10192 rpcz_store.cc:276] 0507 08:05:46.986768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.988354 (+ 1586us) service_pool.cc:225] Handling call
0507 08:05:46.988860 (+ 506us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:46.989163 (+ 303us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:46.989189 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:46.994226 (+ 5037us) log.cc:842] Serialized 564 byte log entry
0507 08:05:47.926519 (+932293us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:47.926558 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:47.926628 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:48.451787 (+525159us) raft_consensus.cc:1671] finished
0507 08:05:48.451815 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:48.451971 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":931997,"threads_started":1,"wal-append.queue_time_us":1124790}
W20240507 08:05:48.382931 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 165289us
I20240507 08:05:48.279619 10185 rpcz_store.cc:276] 0507 08:05:47.113035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.237588 (+124553us) service_pool.cc:225] Handling call
0507 08:05:48.219818 (+982230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:48.406695 10191 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
W20240507 08:05:48.261796 10333 rpcz_store.cc:269] 0507 08:05:46.252969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.274269 (+ 21300us) service_pool.cc:225] Handling call
0507 08:05:48.200510 (+1926241us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b54002c8dd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:48.209857 (+ 9347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":124110592}
I20240507 08:05:48.383546 11628 raft_consensus.cc:686] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:48.393898 11659 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.406059 11654 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:48.408489 11534 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:48.286381 11598 consensus_queue.cc:237] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.421700 11558 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:48.443964 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 482340us
I20240507 08:05:48.446905 11669 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:48.447441 11642 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.425978 11467 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:48.494738 11645 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20240507 08:05:48.428947 11606 raft_consensus.cc:686] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:48.457906 10336 raft_consensus.cc:2354] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:48.432049 11526 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:48.439040 11309 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.443981 11638 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:48.449652 11519 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.501863 11659 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.627094 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:48.561646 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 864) took 1317 ms. Trace:
W20240507 08:05:48.524641 10334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 883) took 1536 ms (client timeout 1624 ms). Trace:
I20240507 08:05:48.624963 11671 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.507642 11600 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.438052 11556 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.429416 11336 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:48.426366 11660 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:48.436334 11655 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.494452 11517 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.638337 10189 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:48.502136 11534 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:48.644830 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:48.465338 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.126066 10192 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:49.126770 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1212) took 1038 ms. Trace:
I20240507 08:05:49.127305 11692 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:49.128032 11692 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:48.573697 11667 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20240507 08:05:48.583989 11674 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:48.638314 10335 rpcz_store.cc:276] 0507 08:05:47.243923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.249206 (+ 5283us) service_pool.cc:225] Handling call
0507 08:05:47.257870 (+ 8664us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:47.278799 (+ 20929us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:05:47.315210 (+ 36411us) log.cc:842] Serialized 14 byte log entry
0507 08:05:47.727969 (+412759us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:47.729295 (+ 1326us) log.cc:842] Serialized 564 byte log entry
0507 08:05:47.729480 (+ 185us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:47.729510 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:47.729616 (+ 106us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:48.561286 (+831670us) raft_consensus.cc:1671] finished
0507 08:05:48.561316 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:48.561527 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33730,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":60427,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":351965,"threads_started":1,"wal-append.queue_time_us":811220}
I20240507 08:05:48.681841 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.132752 10336 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:49.133174 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.133827 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.134411 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:49.135109 10335 raft_consensus.cc:2379] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
W20240507 08:05:49.135188 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 904) took 1406 ms (client timeout 1694 ms). Trace:
W20240507 08:05:49.135567 10191 rpcz_store.cc:269] 0507 08:05:47.728364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.174116 (+445752us) service_pool.cc:225] Handling call
0507 08:05:49.135039 (+960923us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54003b4554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:49.135135 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":509604096}
I20240507 08:05:49.138463 11695 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:49.139161 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1134) took 1171 ms. Trace:
I20240507 08:05:49.138969 11695 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.532202 11558 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.605331 11656 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.599709 11658 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.595773 11581 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:48.574087 10934 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:48.524183 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:48.690265 11556 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:48.585537 11401 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.463589 11627 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.462204 11377 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.149106 11556 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:49.150576 11556 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.151664 11377 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.640460 11638 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:49.152562 11638 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:49.153158 11556 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:48.678848 11534 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:48.573307 11541 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:48.639699 10334 rpcz_store.cc:269] 0507 08:05:46.987953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.993413 (+ 5460us) service_pool.cc:225] Handling call
0507 08:05:48.524344 (+1530931us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b5400049f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:48.524553 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1666291072}
I20240507 08:05:48.553717 11351 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.516287 11467 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.687752 11309 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:48.692454 11650 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.570720 11478 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.642288 11526 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.733065 10189 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:48.550992 11628 consensus_queue.cc:237] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.642192 11671 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.534768 11654 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.664252 11621 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.742321 10906 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:48.735036 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:48.650862 11660 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:48.513319 11669 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.725829 11606 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:48.811103 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:48.654960 11655 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:49.058223 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 883) took 2077 ms (client timeout 1895 ms). Trace:
W20240507 08:05:49.091279 11616 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 1.874s user 0.008s sys 0.008s
I20240507 08:05:49.058939 11519 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.083140 11659 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:49.188409 11669 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:49.117424 11686 log.cc:924] Time spent T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240507 08:05:49.059186 11617 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:49.119654 11555 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:49.190277 11555 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:49.123275 10156 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.038s sys 0.039s
I20240507 08:05:49.173753 10333 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:49.127140 10192 rpcz_store.cc:276] 0507 08:05:48.088314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.454457 (+366143us) service_pool.cc:225] Handling call
0507 08:05:49.126700 (+672243us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:49.193040 10156 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.782s user 0.039s sys 0.040s
I20240507 08:05:49.139531 10266 rpcz_store.cc:276] 0507 08:05:47.967487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.971655 (+ 4168us) service_pool.cc:225] Handling call
0507 08:05:48.044105 (+ 72450us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:48.078448 (+ 34343us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:48.090303 (+ 11855us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:48.093990 (+ 3687us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:48.111109 (+ 17119us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:49.138897 (+1027788us) spinlock_profiling.cc:243] Waited 8.86 ms on lock 0x7b540019d5a0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dacd30 00007fcf90db7973 00007fcf90e2ad92 00007fcf90e2817e 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782
0507 08:05:49.138988 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:49.139109 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20380416}
I20240507 08:05:49.149329 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:05:49.194388 10190 rpcz_store.cc:269] 0507 08:05:46.980467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.982343 (+ 1876us) service_pool.cc:225] Handling call
0507 08:05:49.057967 (+2075624us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000820d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:49.058128 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236617600}
I20240507 08:05:49.155118 11401 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:48.534175 11646 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.157052 11627 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.145635 11661 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:49.222098 11661 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:49.142242 11653 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:49.223074 11467 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:49.168223 10230 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.013s sys 0.026s
I20240507 08:05:49.170065 11692 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:49.170926 11541 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.171861 10267 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:49.148129 10934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43924 (request call id 79) took 1652 ms. Trace:
I20240507 08:05:49.144950 11558 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.169438 11534 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:49.178032 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 897) took 2146 ms (client timeout 1551 ms). Trace:
I20240507 08:05:49.190209 11617 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.034490 11598 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:49.146323 11581 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:49.193327 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 451326us
W20240507 08:05:49.235420 10189 rpcz_store.cc:269] 0507 08:05:47.028031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.053761 (+ 25730us) service_pool.cc:225] Handling call
0507 08:05:48.626881 (+1573120us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:49.174676 (+547795us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3505408}
I20240507 08:05:49.236521 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.237218 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1211) took 1169 ms. Trace:
I20240507 08:05:49.196173 11660 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.239559 10156 leader_election.cc:398] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
W20240507 08:05:49.199221 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 49993us
I20240507 08:05:49.200096 10943 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:49.200651 11655 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:49.200812 11671 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.201723 11478 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:49.198448 11669 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:49.242134 11351 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.241695 10306 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:49.241780 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.242558 11669 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.243167 11655 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:49.243376 10333 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:49.243748 11655 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:49.244280 11655 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:49.244369 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.244738 11669 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.202129 11519 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:49.245357 11519 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:49.246977 11258 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:49.247180 11519 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:49.201020 11654 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:49.178611 11635 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.811s user 0.008s sys 0.007s
W20240507 08:05:49.247802 11635 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.906s user 0.008s sys 0.007s
I20240507 08:05:49.173450 11696 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:49.247946 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.248617 11696 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.205338 11694 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:49.215734 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.221303 10184 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:49.224357 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.166422 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:49.226415 10230 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.753s user 0.014s sys 0.026s
I20240507 08:05:49.250492 10190 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:49.219403 11659 raft_consensus.cc:2781] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:49.198137 11526 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:49.237507 10189 rpcz_store.cc:276] 0507 08:05:48.068131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.236267 (+1168136us) service_pool.cc:225] Handling call
0507 08:05:49.237172 (+ 905us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.240041 10156 leader_election.cc:401] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240507 08:05:49.256275 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.256919 10156 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:49.228873 11695 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:49.206297 11638 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:49.251148 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1197) took 2005 ms. Trace:
I20240507 08:05:49.261899 10190 rpcz_store.cc:276] 0507 08:05:47.246202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.223985 (+1977783us) service_pool.cc:225] Handling call
0507 08:05:49.251078 (+ 27093us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.220723 11555 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:49.254786 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1218) took 1052 ms. Trace:
I20240507 08:05:49.175343 11309 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:49.266101 10192 rpcz_store.cc:276] 0507 08:05:48.202437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.215356 (+1012919us) service_pool.cc:225] Handling call
0507 08:05:49.254731 (+ 39375us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.249020 11581 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:49.257071 10189 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:49.268198 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:49.270121 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1219) took 1014 ms. Trace:
I20240507 08:05:49.270215 10190 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:49.270696 10189 rpcz_store.cc:276] 0507 08:05:48.255879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.255966 (+1000087us) service_pool.cc:225] Handling call
0507 08:05:49.270044 (+ 14078us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.271922 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:49.272953 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1213) took 1155 ms. Trace:
I20240507 08:05:49.273358 10189 rpcz_store.cc:276] 0507 08:05:48.117273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.271539 (+1154266us) service_pool.cc:225] Handling call
0507 08:05:49.272889 (+ 1350us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.274408 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.284039 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:49.342772 10190 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:49.343631 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1207) took 1419 ms. Trace:
I20240507 08:05:49.344018 10190 rpcz_store.cc:276] 0507 08:05:47.924420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.267960 (+1343540us) service_pool.cc:225] Handling call
0507 08:05:49.343556 (+ 75596us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.259312 10185 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:49.354984 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:49.351018 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:49.265911 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:49.233950 10934 rpcz_store.cc:276] 0507 08:05:47.494707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.727533 (+232826us) service_pool.cc:225] Handling call
0507 08:05:49.147252 (+1419719us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:49.265681 11659 raft_consensus.cc:686] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:49.252399 11598 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:49.532568 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 277764us
I20240507 08:05:49.345134 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:49.749117 11692 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:49.925311 11309 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:49.266407 11654 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:49.261328 11660 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:50.219907 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 869859us
I20240507 08:05:49.891069 11258 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:50.235785 10190 raft_consensus.cc:2362] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4. Re-sending same reply.
I20240507 08:05:50.285461 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 51) took 1114 ms. Trace:
I20240507 08:05:50.285935 10190 rpcz_store.cc:276] 0507 08:05:49.171259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.344806 (+173547us) service_pool.cc:225] Handling call
0507 08:05:50.285377 (+940571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.222991 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:50.295325 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 72455.7us
I20240507 08:05:50.296594 10304 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:50.299362 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:50.302760 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:50.303602 10190 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:50.222572 11581 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:50.239509 10189 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
W20240507 08:05:50.323873 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 906) took 2580 ms (client timeout 2929 ms). Trace:
W20240507 08:05:50.324426 10189 rpcz_store.cc:269] 0507 08:05:47.743639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.350755 (+1607116us) service_pool.cc:225] Handling call
0507 08:05:50.323800 (+973045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.313009 11258 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:50.337772 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50099.3us
I20240507 08:05:50.342779 10156 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.357214 10190 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
W20240507 08:05:50.362471 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1216) took 2213 ms (client timeout 2791 ms). Trace:
W20240507 08:05:50.363332 10190 rpcz_store.cc:269] 0507 08:05:48.152104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.302408 (+2150304us) service_pool.cc:225] Handling call
0507 08:05:50.362403 (+ 59995us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.366078 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.367110 10190 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:50.273075 11654 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:50.313689 10935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43924 (request call id 80) took 1075 ms. Trace:
I20240507 08:05:50.269806 11692 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.319792 10333 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:50.386096 11627 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:50.337839 11534 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:50.369706 11699 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:50.351521 11401 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:50.416862 11478 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:50.369484 11695 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:50.460831 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 54) took 1135 ms. Trace:
I20240507 08:05:50.495970 10190 rpcz_store.cc:276] 0507 08:05:49.232890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.365775 (+1132885us) service_pool.cc:225] Handling call
0507 08:05:50.368138 (+ 2363us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.344089 11581 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.329120 10908 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:50.463770 11627 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.473548 11534 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:50.451241 11654 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.518023 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:50.493237 10907 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.078s sys 0.161s
I20240507 08:05:50.481227 11699 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:50.548779 10190 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:50.483438 11558 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.556627 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1224) took 1298 ms. Trace:
I20240507 08:05:50.559563 10190 rpcz_store.cc:276] 0507 08:05:49.258760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.517643 (+1258883us) service_pool.cc:225] Handling call
0507 08:05:50.556567 (+ 38924us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.557026 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.560631 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.561159 10189 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
W20240507 08:05:50.562901 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 263729us
I20240507 08:05:50.438424 10935 rpcz_store.cc:276] 0507 08:05:49.237806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.239816 (+ 2010us) service_pool.cc:225] Handling call
0507 08:05:50.303629 (+1063813us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.563622 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:50.480435 11638 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:50.561903 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1227) took 1186 ms (client timeout 1580 ms). Trace:
W20240507 08:05:50.565003 10306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.305s user 0.046s sys 0.156s
I20240507 08:05:50.480084 11696 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.565958 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:50.570550 10307 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:50.570708 10306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.310s user 0.047s sys 0.156s
I20240507 08:05:50.491619 11621 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.547678 11519 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:50.479669 11617 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.451845 11659 consensus_queue.cc:237] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:50.549273 11627 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.572538 10190 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.485615 11401 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:50.590629 10304 connection.cc:232] RPC call timeout handler was delayed by -1.0247839710000335s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
W20240507 08:05:50.570650 10189 rpcz_store.cc:269] 0507 08:05:49.375399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.325391 (+949992us) service_pool.cc:225] Handling call
0507 08:05:50.561839 (+236448us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:50.574116 10306 connection.cc:232] RPC call timeout handler was delayed by -1.2478656140000339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20240507 08:05:50.572849 11699 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:50.592468 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.554739 11669 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:50.575452 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 264508us
I20240507 08:05:50.562106 11258 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:50.553786 11534 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.591374 10190 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:50.592217 11699 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:50.593302 10189 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
W20240507 08:05:50.597288 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1208) took 2604 ms (client timeout 3324 ms). Trace:
W20240507 08:05:50.597707 10190 rpcz_store.cc:269] 0507 08:05:47.992630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.565732 (+2573102us) service_pool.cc:225] Handling call
0507 08:05:50.597230 (+ 31498us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:50.554566 10907 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.662s user 0.103s sys 0.182s
W20240507 08:05:50.577646 11694 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.110s user 0.011s sys 0.037s
I20240507 08:05:50.549050 11478 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:50.597755 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 53) took 1424 ms. Trace:
I20240507 08:05:50.599125 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.578958 11696 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:50.579895 11598 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.596336 11258 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:50.606434 10189 rpcz_store.cc:276] 0507 08:05:49.173116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.592172 (+1419056us) service_pool.cc:225] Handling call
0507 08:05:50.597607 (+ 5435us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.611428 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.612098 10189 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 1.
I20240507 08:05:50.612605 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 50) took 1441 ms. Trace:
I20240507 08:05:50.612802 10189 rpcz_store.cc:276] 0507 08:05:49.170945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.610737 (+1439792us) service_pool.cc:225] Handling call
0507 08:05:50.612547 (+ 1810us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:50.603996 11694 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.248s user 0.011s sys 0.037s
I20240507 08:05:50.597733 11558 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:50.604856 11655 consensus_meta.cc:287] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 1.360s user 0.014s sys 0.004s
I20240507 08:05:50.606830 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1221) took 1498 ms. Trace:
I20240507 08:05:50.576771 11519 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.604336 11699 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:50.626518 11460 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.627889 11519 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:50.605466 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 713806us
I20240507 08:05:50.615995 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.630177 11692 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.630616 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1223) took 1376 ms. Trace:
I20240507 08:05:50.631119 10189 rpcz_store.cc:276] 0507 08:05:49.253764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.613394 (+1359630us) service_pool.cc:225] Handling call
0507 08:05:50.630436 (+ 17042us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.616487 10190 rpcz_store.cc:276] 0507 08:05:49.108734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.598880 (+1490146us) service_pool.cc:225] Handling call
0507 08:05:50.606770 (+ 7890us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.614501 11694 raft_consensus.cc:686] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:50.618353 11478 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:50.610217 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:50.619850 10267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1136) took 2651 ms (client timeout 2014 ms). Trace:
I20240507 08:05:50.597448 11669 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.626159 11534 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.616242 11696 raft_consensus.cc:686] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:50.614859 11258 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.633961 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.635390 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.624912 11627 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:50.636945 10267 rpcz_store.cc:269] 0507 08:05:47.968288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.968442 (+ 154us) service_pool.cc:225] Handling call
0507 08:05:50.619569 (+2651127us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002909d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:50.619747 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2538717952}
I20240507 08:05:50.646118 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1230) took 1049 ms. Trace:
I20240507 08:05:50.647691 10190 rpcz_store.cc:276] 0507 08:05:50.223160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.634970 (+411810us) service_pool.cc:225] Handling call
0507 08:05:50.646073 (+ 11103us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.647423 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1226) took 1371 ms. Trace:
W20240507 08:05:50.638214 11705 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20240507 08:05:50.648339 10189 rpcz_store.cc:276] 0507 08:05:49.275942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.632365 (+1356423us) service_pool.cc:225] Handling call
0507 08:05:50.647350 (+ 14985us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.649210 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.617050 11558 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:50.657155 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1229) took 1061 ms. Trace:
I20240507 08:05:50.658182 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.658412 10190 rpcz_store.cc:276] 0507 08:05:49.595859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.649015 (+1053156us) service_pool.cc:225] Handling call
0507 08:05:50.657078 (+ 8063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.659010 10189 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:50.632922 11655 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.662461 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:50.663288 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 71156.3us
I20240507 08:05:50.655114 11581 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:50.664444 10306 connection.cc:232] RPC call timeout handler was delayed by -1.1620984219999855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20240507 08:05:50.664875 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.664098 10190 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
W20240507 08:05:50.665329 10306 connection.cc:232] RPC call timeout handler was delayed by -1.0292587040000285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
I20240507 08:05:50.660180 11401 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.667728 10306 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:50.669131 10306 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.669427 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 890) took 3682 ms. Trace:
I20240507 08:05:50.670218 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.670810 10189 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:50.673708 11669 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.675805 11617 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:50.676208 11617 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:50.676527 11617 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.678038 11660 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
W20240507 08:05:50.664546 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 372072us
I20240507 08:05:50.678627 11699 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:50.679286 10908 leader_election.cc:334] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.764s (SENT)
W20240507 08:05:50.680426 11704 log.cc:924] Time spent T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 1.085s user 0.000s sys 0.003s
I20240507 08:05:50.680346 10908 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:50.680914 11699 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.669823 10190 rpcz_store.cc:276] 0507 08:05:46.987278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.668332 (+3681054us) service_pool.cc:225] Handling call
0507 08:05:50.669007 (+ 675us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.669149 (+ 142us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.669172 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.669210 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:50.669239 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.669288 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.669381 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.682767 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 895) took 3655 ms. Trace:
I20240507 08:05:50.682950 11627 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.667376 11712 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:50.683601 11581 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:50.667520 11710 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.683655 11617 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.684644 11531 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.684612 11710 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:50.686300 11617 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.687009 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:50.688077 10185 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:50.688374 11258 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:50.688627 10185 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1210) took 2633 ms (client timeout 1750 ms). Trace:
I20240507 08:05:50.689565 11659 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:50.689966 11659 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.690306 11659 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.690852 11409 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.691069 11558 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:50.684129 11581 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:50.660703 11694 consensus_queue.cc:237] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.694103 11581 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:50.665964 11696 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:50.694378 11710 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:50.697063 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:50.683137 10190 rpcz_store.cc:276] 0507 08:05:47.027493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.682130 (+3654637us) service_pool.cc:225] Handling call
0507 08:05:50.682369 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.682504 (+ 135us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.682533 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.682567 (+ 34us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:50.682593 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.682641 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.682721 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:50.698817 10943 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43936 (request call id 87) took 1959 ms (client timeout 2100 ms). Trace:
I20240507 08:05:50.699039 11556 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:50.699705 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 896) took 3672 ms. Trace:
I20240507 08:05:50.699546 11556 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.688592 11617 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:50.700078 10190 rpcz_store.cc:276] 0507 08:05:47.027786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.699003 (+3671217us) service_pool.cc:225] Handling call
0507 08:05:50.699262 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.699398 (+ 136us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.699423 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.699459 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:50.699487 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.699548 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.699654 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.700906 11617 raft_consensus.cc:686] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:50.688330 10306 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.703191 10306 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.704432 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 898) took 3676 ms. Trace:
W20240507 08:05:50.689378 10185 rpcz_store.cc:269] 0507 08:05:48.055034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.166130 (+1111096us) service_pool.cc:225] Handling call
0507 08:05:50.688564 (+1522434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.705060 11710 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:50.705480 11710 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.702461 11617 consensus_queue.cc:237] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.700249 11655 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.705932 10185 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:50.706296 10265 raft_consensus.cc:2379] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:50.706571 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 902) took 3552 ms. Trace:
I20240507 08:05:50.706971 10185 rpcz_store.cc:276] 0507 08:05:47.154267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.705340 (+3551073us) service_pool.cc:225] Handling call
0507 08:05:50.705633 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.706357 (+ 724us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.706527 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.706985 10332 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:50.705994 11710 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:50.710644 10190 rpcz_store.cc:276] 0507 08:05:47.028318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.703580 (+3675262us) service_pool.cc:225] Handling call
0507 08:05:50.703988 (+ 408us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.704135 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.704160 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.704201 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:50.704233 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.704284 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.704382 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.711694 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:50.712167 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1194) took 3466 ms. Trace:
I20240507 08:05:50.712302 10935 raft_consensus.cc:1799] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Handling vote request from an unknown peer 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:50.712973 10935 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240507 08:05:50.683959 11713 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:05:50.699234 10943 rpcz_store.cc:269] 0507 08:05:48.739748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.740741 (+ 993us) service_pool.cc:225] Handling call
0507 08:05:50.698623 (+1957882us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b54005113d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:50.698762 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5318400}
I20240507 08:05:50.713376 10262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 895) took 2702 ms. Trace:
I20240507 08:05:50.712575 10190 rpcz_store.cc:276] 0507 08:05:47.245393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.711517 (+3466124us) service_pool.cc:225] Handling call
0507 08:05:50.711747 (+ 230us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.711894 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.711916 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.711954 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:50.711981 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.712029 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.712121 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.717819 10262 rpcz_store.cc:276] 0507 08:05:48.011435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.150982 (+139547us) service_pool.cc:225] Handling call
0507 08:05:48.173604 (+ 22622us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:48.173779 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:48.173806 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:48.173846 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:48.173874 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:48.173927 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.713197 (+2539270us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b54002ad0d0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:50.713306 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51223424}
I20240507 08:05:50.719576 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1195) took 3474 ms. Trace:
I20240507 08:05:50.718020 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 58) took 2258 ms. Trace:
I20240507 08:05:50.719906 10190 rpcz_store.cc:276] 0507 08:05:47.245679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.718885 (+3473206us) service_pool.cc:225] Handling call
0507 08:05:50.719157 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.719304 (+ 147us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.719327 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.719365 (+ 38us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:50.719392 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.719439 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.719532 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.721046 10306 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.683701 11712 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.706254 11409 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:50.721558 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1198) took 3475 ms. Trace:
I20240507 08:05:50.732160 11409 raft_consensus.cc:686] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:50.732038 10190 rpcz_store.cc:276] 0507 08:05:47.246469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.720919 (+3474450us) service_pool.cc:225] Handling call
0507 08:05:50.721162 (+ 243us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.721302 (+ 140us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.721324 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.721359 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:50.721383 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.721427 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.721515 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.724754 11712 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:50.736304 10231 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.309s user 0.001s sys 0.031s
I20240507 08:05:50.735903 11659 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:50.721841 11530 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:50.736433 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1203) took 3438 ms. Trace:
I20240507 08:05:50.726272 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.737807 10306 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.736882 11530 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.733351 11409 consensus_queue.cc:237] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.739713 11530 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:50.736892 10190 rpcz_store.cc:276] 0507 08:05:47.298110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.733726 (+3435616us) service_pool.cc:225] Handling call
0507 08:05:50.734522 (+ 796us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.734653 (+ 131us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.734680 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.734746 (+ 66us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:50.734781 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.734827 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.736373 (+ 1546us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.763769 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1205) took 3208 ms. Trace:
I20240507 08:05:50.764114 10190 rpcz_store.cc:276] 0507 08:05:47.555239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.762947 (+3207708us) service_pool.cc:225] Handling call
0507 08:05:50.763262 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.763427 (+ 165us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.763458 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.763509 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:50.763545 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.763600 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.763714 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.767251 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.768102 10186 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
W20240507 08:05:50.768529 10186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 884) took 3787 ms (client timeout 2284 ms). Trace:
W20240507 08:05:50.768867 10186 rpcz_store.cc:269] 0507 08:05:46.980779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.983394 (+ 2615us) service_pool.cc:225] Handling call
0507 08:05:50.767085 (+3783691us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:50.768451 (+ 1366us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6015744}
I20240507 08:05:50.740540 11712 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:50.725523 10334 rpcz_store.cc:276] 0507 08:05:48.460020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.176803 (+716783us) service_pool.cc:225] Handling call
0507 08:05:49.177109 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:49.227301 (+ 50192us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:49.228919 (+ 1618us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:49.233329 (+ 4410us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:49.235126 (+ 1797us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.717796 (+1482670us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b5400284454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:50.717847 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.717970 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39438208}
I20240507 08:05:50.769765 11712 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.770161 11712 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:50.770366 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:50.771157 10334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 917) took 1571 ms (client timeout 1765 ms). Trace:
I20240507 08:05:50.771046 11530 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:50.771706 10334 rpcz_store.cc:269] 0507 08:05:49.200216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.770107 (+1569891us) service_pool.cc:225] Handling call
0507 08:05:50.771100 (+ 993us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.772756 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:50.742331 11655 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:50.773481 11655 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:50.774441 11655 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
W20240507 08:05:50.742693 10231 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.313s user 0.003s sys 0.031s
W20240507 08:05:50.775822 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 2.35214e+06us
I20240507 08:05:50.776178 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1206) took 3034 ms. Trace:
I20240507 08:05:50.777000 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 892) took 3782 ms. Trace:
I20240507 08:05:50.746668 11710 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.725033 11654 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.751845 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 904) took 2353 ms. Trace:
I20240507 08:05:50.755352 11581 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:50.689402 10908 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
W20240507 08:05:50.784791 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 95518.3us
I20240507 08:05:50.785526 10908 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:50.786310 11581 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.761505 10306 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:50.786341 11713 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:50.787057 11713 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.787462 11713 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:50.788194 11654 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.789744 11623 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:50.790107 11710 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.762162 11659 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.777384 10231 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:50.791245 10192 raft_consensus.cc:2379] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:50.776527 10190 rpcz_store.cc:276] 0507 08:05:47.743356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.765034 (+3021678us) service_pool.cc:225] Handling call
0507 08:05:50.765306 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.765468 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.765508 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.765550 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:50.775969 (+ 10419us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.776031 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.776133 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.800691 11655 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.801649 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1209) took 2756 ms. Trace:
I20240507 08:05:50.802064 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.802018 10186 rpcz_store.cc:276] 0507 08:05:48.044829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.776736 (+2731907us) service_pool.cc:225] Handling call
0507 08:05:50.800464 (+ 23728us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.800650 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.800675 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.800715 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:50.800749 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.800802 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.800907 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:50.803437 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.803848 11655 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.803584 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:50.804543 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:50.783921 11712 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.787483 11409 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:50.804955 11661 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:50.805512 11409 raft_consensus.cc:686] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:50.806560 11712 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.806329 11409 consensus_queue.cc:237] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:50.804992 10190 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:50.790127 11623 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:50.809971 11623 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:50.810433 11556 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.794104 11711 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:50.780261 11530 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:50.781011 10335 rpcz_store.cc:276] 0507 08:05:48.398765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.143370 (+744605us) service_pool.cc:225] Handling call
0507 08:05:49.143705 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:49.259607 (+115902us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:49.259654 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:49.259711 (+ 57us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:49.259753 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.751529 (+1491776us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540034e4d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:50.751595 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.751770 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":266214144}
W20240507 08:05:52.643574 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 922) took 3442 ms (client timeout 1890 ms). Trace:
W20240507 08:05:52.643952 10335 rpcz_store.cc:269] 0507 08:05:49.201486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.643373 (+3441887us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.643486 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.644630 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 920) took 3443 ms (client timeout 2002 ms). Trace:
W20240507 08:05:52.644817 10335 rpcz_store.cc:269] 0507 08:05:49.200975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.644509 (+3443534us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.644575 (+ 66us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.645228 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 912) took 3446 ms (client timeout 2065 ms). Trace:
I20240507 08:05:50.820029 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1193) took 3575 ms. Trace:
W20240507 08:05:52.688669 9051 debug-util.cc:398] Leaking SignalData structure 0x7b080000ed40 after lost signal to thread 10231
I20240507 08:05:52.561393 10090 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 3 to 4, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 1 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:05:52.708395 9051 kernel_stack_watchdog.cc:198] Thread 10231 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 893ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20240507 08:05:50.794514 10231 leader_election.cc:398] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
W20240507 08:05:51.972549 10156 leader_election.cc:334] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.624s (SENT)
W20240507 08:05:51.061564 10158 connection.cc:232] RPC call timeout handler was delayed by -1.1321895180000183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
I20240507 08:05:52.712385 10231 leader_election.cc:401] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:05:52.718039 11538 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
W20240507 08:05:52.719166 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 1.9277e+06us
I20240507 08:05:50.904206 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:50.815099 10190 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
W20240507 08:05:52.727475 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1241) took 1932 ms (client timeout 1699 ms). Trace:
W20240507 08:05:52.727890 10190 rpcz_store.cc:269] 0507 08:05:50.794926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.803339 (+ 8413us) service_pool.cc:225] Handling call
0507 08:05:52.727194 (+1923855us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:50.920276 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1217) took 2728 ms (client timeout 1666 ms). Trace:
W20240507 08:05:52.731024 10191 rpcz_store.cc:269] 0507 08:05:48.191869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.920046 (+2728177us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:50.920177 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.732234 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1245) took 1745 ms (client timeout 1518 ms). Trace:
W20240507 08:05:52.732623 10191 rpcz_store.cc:269] 0507 08:05:50.986549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.732001 (+1745452us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.732163 (+ 162us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:50.819164 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:52.735193 10186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1237) took 1942 ms (client timeout 1849 ms). Trace:
W20240507 08:05:52.735572 10186 rpcz_store.cc:269] 0507 08:05:50.792622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.818915 (+ 26293us) service_pool.cc:225] Handling call
0507 08:05:52.735026 (+1916111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.736407 10186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1250) took 1748 ms (client timeout 1633 ms). Trace:
W20240507 08:05:52.736728 10186 rpcz_store.cc:269] 0507 08:05:50.988130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.736215 (+1748085us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.736363 (+ 148us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:51.043174 10184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1196) took 3797 ms (client timeout 2014 ms). Trace:
W20240507 08:05:52.738848 10184 rpcz_store.cc:269] 0507 08:05:47.245932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.090202 (+844270us) service_pool.cc:225] Handling call
0507 08:05:51.042948 (+2952746us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001be9d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:51.043105 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2352975744}
W20240507 08:05:52.740252 10184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 56) took 1946 ms (client timeout 1885 ms). Trace:
W20240507 08:05:52.740628 10184 rpcz_store.cc:269] 0507 08:05:50.793851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.740043 (+1946192us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.740171 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:50.777339 10189 rpcz_store.cc:276] 0507 08:05:46.994916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.671725 (+3676809us) service_pool.cc:225] Handling call
0507 08:05:50.672051 (+ 326us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:50.672247 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:50.672273 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:50.672693 (+ 420us) log.cc:842] Serialized 26 byte log entry
0507 08:05:50.756559 (+ 83866us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:50.756597 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.756661 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:50.776823 (+ 20162us) raft_consensus.cc:1671] finished
0507 08:05:50.776841 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.776951 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":83497,"threads_started":1,"wal-append.queue_time_us":84414}
I20240507 08:05:50.987457 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:51.015972 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:52.270810 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2140 after lost signal to thread 9046
W20240507 08:05:50.844141 10232 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.351s user 0.034s sys 0.043s
W20240507 08:05:50.896723 10233 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.471s user 0.013s sys 0.040s
W20240507 08:05:51.255983 10192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1220) took 2937 ms (client timeout 1887 ms). Trace:
W20240507 08:05:52.746551 10188 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 903) took 5018 ms (client timeout 1631 ms). Trace:
W20240507 08:05:50.999430 10261 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.558s user 0.025s sys 0.024s
W20240507 08:05:52.755736 10192 rpcz_store.cc:269] 0507 08:05:48.318322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.266833 (+948511us) service_pool.cc:225] Handling call
0507 08:05:51.255744 (+1988911us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54000820d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:51.255911 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1310984960}
I20240507 08:05:50.931772 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1141) took 2933 ms. Trace:
I20240507 08:05:52.760164 10264 rpcz_store.cc:276] 0507 08:05:47.998331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.999475 (+ 1144us) service_pool.cc:225] Handling call
0507 08:05:48.007289 (+ 7814us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:48.101232 (+ 93943us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:48.101275 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:48.101330 (+ 55us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:48.101371 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.931588 (+2830217us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b54002be654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:50.931624 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.931729 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":215512448}
W20240507 08:05:52.521400 10288 debug-util.cc:398] Leaking SignalData structure 0x7b080016dd40 after lost signal to thread 9046
I20240507 08:05:50.988931 10306 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:51.084242 10334 raft_consensus.cc:2379] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:51.120551 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 891) took 3119 ms. Trace:
I20240507 08:05:51.166878 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:52.769021 10258 rpcz_store.cc:276] 0507 08:05:48.001003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.130977 (+129974us) service_pool.cc:225] Handling call
0507 08:05:48.131795 (+ 818us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:48.133182 (+ 1387us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:48.133220 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:48.149434 (+ 16214us) log.cc:842] Serialized 26 byte log entry
0507 08:05:48.279789 (+130355us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:48.279831 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:48.279903 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:51.120322 (+2840419us) raft_consensus.cc:1671] finished
0507 08:05:51.120337 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:51.120488 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1110,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":4404,"thread_start_us":101269,"threads_started":1,"wal-append.queue_time_us":270523,"wal-append.run_cpu_time_us":17530,"wal-append.run_wall_time_us":1693152}
W20240507 08:05:51.100214 10265 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 906) took 2927 ms (client timeout 2018 ms). Trace:
W20240507 08:05:52.627485 10332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 905) took 4228 ms (client timeout 1577 ms). Trace:
W20240507 08:05:52.773597 10265 rpcz_store.cc:269] 0507 08:05:48.172827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.197966 (+ 25139us) service_pool.cc:225] Handling call
0507 08:05:51.099959 (+2901993us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b5400117454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:51.100127 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5297750912}
W20240507 08:05:52.774000 10332 rpcz_store.cc:269] 0507 08:05:48.399036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.439534 (+ 40498us) service_pool.cc:225] Handling call
0507 08:05:52.627271 (+4187737us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b540004ced4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:52.627408 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5215111040}
W20240507 08:05:52.775787 10332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 921) took 3574 ms (client timeout 2123 ms). Trace:
W20240507 08:05:52.776134 10332 rpcz_store.cc:269] 0507 08:05:49.201242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.775512 (+3574270us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.775706 (+ 194us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.776965 10332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 914) took 3577 ms (client timeout 2198 ms). Trace:
I20240507 08:05:51.205386 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 870) took 4424 ms. Trace:
W20240507 08:05:52.631167 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 289873us
I20240507 08:05:52.778626 10340 rpcz_store.cc:276] 0507 08:05:46.780811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.790468 (+ 9657us) service_pool.cc:225] Handling call
0507 08:05:46.841141 (+ 50673us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:47.974246 (+1133105us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:51.205115 (+3230869us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54003c4e54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:51.205313 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2119392512}
I20240507 08:05:51.184590 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 867) took 3940 ms. Trace:
I20240507 08:05:52.782382 10341 rpcz_store.cc:276] 0507 08:05:47.244631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.255083 (+ 10452us) service_pool.cc:225] Handling call
0507 08:05:47.255339 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:47.761695 (+506356us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:47.761747 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:47.761789 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:47.761827 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:51.184399 (+3422572us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400283f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:51.184448 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:51.184548 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1164626688}
W20240507 08:05:52.783900 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 924) took 2434 ms (client timeout 1588 ms). Trace:
W20240507 08:05:52.784287 10341 rpcz_store.cc:269] 0507 08:05:50.349989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.783713 (+2433724us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.783845 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.785070 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 927) took 2434 ms (client timeout 1624 ms). Trace:
I20240507 08:05:52.783100 10086 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59936 (request call id 156) took 8757 ms. Trace:
W20240507 08:05:51.178850 10336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 911) took 1980 ms (client timeout 1685 ms). Trace:
I20240507 08:05:51.227418 10939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43910 (request call id 139) took 2948 ms. Trace:
I20240507 08:05:52.602319 10935 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240507 08:05:51.065505 11601 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:52.785379 10341 rpcz_store.cc:269] 0507 08:05:50.350865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.784898 (+2434033us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.785030 (+ 132us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.785604 10086 rpcz_store.cc:276] 0507 08:05:44.025366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:44.025531 (+ 165us) service_pool.cc:225] Handling call
0507 08:05:52.782827 (+8757296us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:45.504229 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:45.696175 (+191946us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:45.698630 (+ 2455us) write_op.cc:435] Acquired schema lock
0507 08:05:45.698656 (+ 26us) tablet.cc:660] Decoding operations
0507 08:05:45.958382 (+259726us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:45.958445 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:45.958474 (+ 29us) tablet.cc:683] Acquiring locks for 6 operations
0507 08:05:45.959019 (+ 545us) tablet.cc:699] Row locks acquired
0507 08:05:45.959037 (+ 18us) write_op.cc:260] PREPARE: finished
0507 08:05:45.959189 (+ 152us) write_op.cc:270] Start()
0507 08:05:45.959300 (+ 111us) write_op.cc:276] Timestamp: P: 1715069145959144 usec, L: 0
0507 08:05:45.959325 (+ 25us) op_driver.cc:331] REPLICATION: starting
0507 08:05:45.960007 (+ 682us) log.cc:842] Serialized 2481 byte log entry
0507 08:05:47.287222 (+1327215us) op_driver.cc:444] REPLICATION: finished
0507 08:05:49.163728 (+1876506us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf8dedf16e 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf94838dd0
0507 08:05:49.179042 (+ 15314us) write_op.cc:301] APPLY: starting
0507 08:05:49.179123 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:49.179292 (+ 169us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:49.179313 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:49.180061 (+ 748us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:49.180241 (+ 180us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0507 08:05:49.180286 (+ 45us) write_op.cc:312] APPLY: finished
0507 08:05:49.180531 (+ 245us) log.cc:842] Serialized 52 byte log entry
0507 08:05:49.259469 (+ 78938us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:49.269333 (+ 9864us) write_op.cc:454] Released schema lock
0507 08:05:49.277816 (+ 8483us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:50.271967 (+994151us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:50.284131 (+ 12164us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2592,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":1859359,"apply.run_cpu_time_us":64168,"apply.run_wall_time_us":1157890,"num_ops":6,"prepare.queue_time_us":341637,"prepare.run_cpu_time_us":61457,"prepare.run_wall_time_us":1432949,"raft.queue_time_us":424021,"raft.run_cpu_time_us":16788,"raft.run_wall_time_us":2013135,"replication_time_us":1299876,"spinlock_wait_cycles":6851456,"thread_start_us":3457924,"threads_started":5,"wal-append.queue_time_us":887357,"wal-append.run_cpu_time_us":44319,"wal-append.run_wall_time_us":2401164}]]}
I20240507 08:05:52.791141 11601 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:52.792104 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 926) took 2441 ms (client timeout 1722 ms). Trace:
W20240507 08:05:52.792505 10341 rpcz_store.cc:269] 0507 08:05:50.350573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.791919 (+2441346us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.792049 (+ 130us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:50.916776 11623 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:51.007239 11654 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:51.206250 10338 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 52) took 4448 ms (client timeout 3166 ms). Trace:
I20240507 08:05:52.799253 11623 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:52.799042 11654 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:52.799651 10338 rpcz_store.cc:269] 0507 08:05:46.757700 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.758060 (+ 360us) service_pool.cc:225] Handling call
0507 08:05:51.206049 (+4447989us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b540013f454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:51.206162 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1221862656}
W20240507 08:05:52.800455 10304 leader_election.cc:334] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 2.014s (SENT)
W20240507 08:05:52.801218 10338 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 880) took 2209 ms (client timeout 1694 ms). Trace:
I20240507 08:05:52.801394 11654 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:52.801792 11601 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:52.801689 10338 rpcz_store.cc:269] 0507 08:05:50.591355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.800984 (+2209629us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.801153 (+ 169us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:51.675065 11530 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:05:52.803416 10338 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 934) took 2120 ms (client timeout 1621 ms). Trace:
W20240507 08:05:52.803912 10338 rpcz_store.cc:269] 0507 08:05:50.682810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.803168 (+2120358us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.803345 (+ 177us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.803558 11530 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:52.805275 10338 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 925) took 2455 ms (client timeout 1983 ms). Trace:
W20240507 08:05:52.805639 10338 rpcz_store.cc:269] 0507 08:05:50.350305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.805090 (+2454785us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.805218 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.806740 11530 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:50.811226 11711 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:52.810190 11711 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:52.812476 10090 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 66) took 7823 ms. Trace:
I20240507 08:05:52.600731 11699 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:52.813426 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f47233c3be4fe99e097ade2aaec1e8 with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:39708
I20240507 08:05:52.814296 11699 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:52.816695 11699 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:52.818768 10088 catalog_manager.cc:5484] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 2 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:52.821125 10088 catalog_manager.cc:5484] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 2 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:05:51.128602 10943 raft_consensus.cc:1226] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:52.827958 10943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 85) took 1700 ms. Trace:
I20240507 08:05:52.828337 10943 rpcz_store.cc:276] 0507 08:05:51.127891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:51.128026 (+ 135us) service_pool.cc:225] Handling call
0507 08:05:51.128397 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:52.827399 (+1699002us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:52.827794 (+ 395us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:51.139246 10938 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 78) took 4466 ms. Trace:
I20240507 08:05:52.830444 10938 rpcz_store.cc:276] 0507 08:05:46.673103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.689524 (+ 16421us) service_pool.cc:225] Handling call
0507 08:05:46.746268 (+ 56744us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:47.149909 (+403641us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:51.139008 (+3989099us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b5400512554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:51.139180 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":710123264}
I20240507 08:05:51.176939 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 866) took 3932 ms. Trace:
I20240507 08:05:52.622344 11625 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:52.832937 11625 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:52.812916 10090 rpcz_store.cc:276] 0507 08:05:44.988883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:44.989016 (+ 133us) service_pool.cc:225] Handling call
0507 08:05:52.812284 (+7823268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:52.773783 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:52.774429 (+ 646us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:52.774478 (+ 49us) write_op.cc:435] Acquired schema lock
0507 08:05:52.774510 (+ 32us) tablet.cc:660] Decoding operations
0507 08:05:52.775027 (+ 517us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:52.775147 (+ 120us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:52.775182 (+ 35us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:05:52.775611 (+ 429us) tablet.cc:699] Row locks acquired
0507 08:05:52.775643 (+ 32us) write_op.cc:260] PREPARE: finished
0507 08:05:52.775927 (+ 284us) write_op.cc:270] Start()
0507 08:05:52.776064 (+ 137us) write_op.cc:276] Timestamp: P: 1715069152775801 usec, L: 0
0507 08:05:52.776098 (+ 34us) op_driver.cc:331] REPLICATION: starting
0507 08:05:52.776895 (+ 797us) log.cc:842] Serialized 535 byte log entry
0507 08:05:52.798116 (+ 21221us) op_driver.cc:444] REPLICATION: finished
0507 08:05:52.806041 (+ 7925us) write_op.cc:301] APPLY: starting
0507 08:05:52.806133 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:52.806522 (+ 389us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:52.806654 (+ 132us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:52.807019 (+ 365us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:52.807307 (+ 288us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:05:52.807394 (+ 87us) write_op.cc:312] APPLY: finished
0507 08:05:52.807708 (+ 314us) log.cc:842] Serialized 22 byte log entry
0507 08:05:52.808511 (+ 803us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:52.808603 (+ 92us) write_op.cc:454] Released schema lock
0507 08:05:52.808663 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:52.808978 (+ 315us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:52.809017 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1247621,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":7602,"apply.run_cpu_time_us":2394,"apply.run_wall_time_us":3224,"num_ops":1,"prepare.queue_time_us":11525,"prepare.run_cpu_time_us":4460,"prepare.run_wall_time_us":4464,"raft.queue_time_us":974,"raft.run_cpu_time_us":7053,"raft.run_wall_time_us":8461,"replication_time_us":21935,"thread_start_us":20004,"threads_started":4,"wal-append.queue_time_us":1145}]]}
I20240507 08:05:50.818116 11713 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:52.832386 10339 rpcz_store.cc:276] 0507 08:05:47.244388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.251662 (+ 7274us) service_pool.cc:225] Handling call
0507 08:05:47.303217 (+ 51555us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:48.090921 (+787704us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:48.090966 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:48.091456 (+ 490us) log.cc:842] Serialized 26 byte log entry
0507 08:05:48.408383 (+316927us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:48.408424 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:51.176586 (+2768162us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54002c83d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:51.176636 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:51.176695 (+ 59us) raft_consensus.cc:1671] finished
0507 08:05:51.176704 (+ 9us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:51.176894 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3207,"raft.queue_time_us":6781,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":7185,"spinlock_wait_cycles":1804079872,"thread_start_us":284406,"threads_started":1,"wal-append.queue_time_us":413659,"wal-append.run_cpu_time_us":9472,"wal-append.run_wall_time_us":1694949}
W20240507 08:05:52.645399 10335 rpcz_store.cc:269] 0507 08:05:49.198845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.645140 (+3446295us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.645200 (+ 60us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.838371 10339 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 935) took 1773 ms (client timeout 1547 ms). Trace:
W20240507 08:05:52.838842 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 936) took 1772 ms (client timeout 1656 ms). Trace:
W20240507 08:05:52.839327 10335 rpcz_store.cc:269] 0507 08:05:51.066097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.838555 (+1772458us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.838695 (+ 140us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:52.840229 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 937) took 1774 ms (client timeout 1685 ms). Trace:
W20240507 08:05:52.840536 10335 rpcz_store.cc:269] 0507 08:05:51.066264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.840063 (+1773799us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.840181 (+ 118us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.841308 11625 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:52.583642 10906 leader_election.cc:334] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.916s (SENT)
W20240507 08:05:52.650070 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 909) took 3452 ms (client timeout 1656 ms). Trace:
W20240507 08:05:52.842614 10333 rpcz_store.cc:269] 0507 08:05:49.198077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.245406 (+ 47329us) service_pool.cc:225] Handling call
0507 08:05:52.649875 (+3404469us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400284954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:52.650004 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2404032256}
W20240507 08:05:52.843755 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 260144us
I20240507 08:05:52.844115 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.845274 10333 raft_consensus.cc:2379] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
W20240507 08:05:52.845739 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 939) took 1779 ms (client timeout 1833 ms). Trace:
I20240507 08:05:50.902853 11653 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:52.848278 11653 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:52.841311 10335 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 932) took 2179 ms (client timeout 2162 ms). Trace:
I20240507 08:05:51.174217 11533 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:52.851634 11699 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:52.852023 11533 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:52.236754 11712 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:52.854233 11712 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:52.854899 11530 consensus_queue.cc:1046] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:52.855851 11530 consensus_queue.cc:1225] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer 5d6656a616b747d393ff3de5328d2540 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240507 08:05:52.856611 11712 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:52.856562 11530 consensus_queue.cc:1046] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:52.858168 11696 raft_consensus.cc:3032] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:52.858671 11696 raft_consensus.cc:729] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:52.860227 11696 consensus_queue.cc:260] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:52.846139 10333 rpcz_store.cc:269] 0507 08:05:51.066660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.843703 (+1777043us) service_pool.cc:225] Handling call
0507 08:05:52.845667 (+ 1964us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:51.070130 11648 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:52.838735 10339 rpcz_store.cc:269] 0507 08:05:51.065055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.838189 (+1773134us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.838319 (+ 130us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20240507 08:05:51.160554 10155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.749s user 0.021s sys 0.035s
I20240507 08:05:52.862890 11648 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:52.864104 10155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.453s user 0.021s sys 0.035s
W20240507 08:05:52.864567 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 4.45943e+06us
I20240507 08:05:52.710275 10185 rpcz_store.cc:276] 0507 08:05:47.245110 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.707848 (+3462738us) service_pool.cc:225] Handling call
0507 08:05:50.708135 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:50.708304 (+ 169us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:50.757284 (+ 48980us) log.cc:842] Serialized 14 byte log entry
0507 08:05:50.819721 (+ 62437us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:50.819770 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:50.819820 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:50.819880 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:50.819989 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":51111,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":99465,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":70,"thread_start_us":110559,"threads_started":2,"wal-append.queue_time_us":47338,"wal-append.run_cpu_time_us":1036,"wal-append.run_wall_time_us":1001057}
I20240507 08:05:52.866353 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.867101 10185 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:52.713124 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 740812us
I20240507 08:05:52.721351 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:52.872418 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 151450us
I20240507 08:05:52.865324 11648 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:52.873028 11653 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:52.737519 10186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1249) took 1749 ms (client timeout 1686 ms). Trace:
I20240507 08:05:52.874131 11538 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20240507 08:05:52.874075 10186 rpcz_store.cc:269] 0507 08:05:50.987889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.737368 (+1749479us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.737477 (+ 109us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.874652 11538 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:52.875133 11538 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:52.875344 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.876149 10186 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:52.876550 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:52.745204 10183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 885) took 5764 ms (client timeout 1748 ms). Trace:
W20240507 08:05:52.885116 10183 rpcz_store.cc:269] 0507 08:05:46.981187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:46.983491 (+ 2304us) service_pool.cc:225] Handling call
0507 08:05:50.987282 (+4003791us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:52.745016 (+1757734us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13584768}
I20240507 08:05:52.886458 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.887312 10183 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:52.755014 10233 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.330s user 0.013s sys 0.042s
W20240507 08:05:52.747038 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8040 after lost signal to thread 9049
W20240507 08:05:52.755695 10188 rpcz_store.cc:269] 0507 08:05:47.728075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.120416 (+392341us) service_pool.cc:225] Handling call
0507 08:05:51.015747 (+2895331us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:52.746334 (+1730587us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2614016}
I20240507 08:05:52.890559 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:52.752622 10232 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 4.260s user 0.038s sys 0.043s
W20240507 08:05:52.890856 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08000e1980 after lost signal to thread 10124
W20240507 08:05:52.891434 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08000e18c0 after lost signal to thread 10138
I20240507 08:05:52.891448 10188 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:52.892325 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 57) took 1838 ms. Trace:
I20240507 08:05:52.892704 10188 rpcz_store.cc:276] 0507 08:05:51.054349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.890212 (+1835863us) service_pool.cc:225] Handling call
0507 08:05:52.892242 (+ 2030us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:52.894032 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.895226 10188 raft_consensus.cc:2396] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:52.895812 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1236) took 2103 ms. Trace:
W20240507 08:05:52.895869 10232 connection.cc:232] RPC call timeout handler was delayed by -4.33927120300001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1748s for a response
I20240507 08:05:52.896251 10188 rpcz_store.cc:276] 0507 08:05:50.792465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.893536 (+2101071us) service_pool.cc:225] Handling call
0507 08:05:52.895752 (+ 2216us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.758678 10261 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 4.318s user 0.025s sys 0.028s
I20240507 08:05:52.757530 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.758719 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:52.762934 10288 debug-util.cc:398] Leaking SignalData structure 0x7b080016dd60 after lost signal to thread 9049
W20240507 08:05:52.764742 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 1.77647e+06us
I20240507 08:05:52.767678 11695 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:52.771250 10259 raft_consensus.cc:2354] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
W20240507 08:05:50.925750 10337 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 871) took 3669 ms (client timeout 1846 ms). Trace:
I20240507 08:05:52.901262 10908 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
W20240507 08:05:52.901728 10337 rpcz_store.cc:269] 0507 08:05:47.256792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.283050 (+ 26258us) service_pool.cc:225] Handling call
0507 08:05:50.925532 (+3642482us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b5400283054. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:50.925686 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1000970496}
I20240507 08:05:52.903487 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.903594 10192 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:52.903676 11625 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:52.904295 10337 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
W20240507 08:05:52.904695 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1243) took 1918 ms (client timeout 1952 ms). Trace:
I20240507 08:05:52.771409 10187 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
W20240507 08:05:52.905290 10190 rpcz_store.cc:269] 0507 08:05:50.985966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.754507 (+1768541us) service_pool.cc:225] Handling call
0507 08:05:52.904618 (+150111us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.905759 10187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1200) took 5658 ms (client timeout 1952 ms). Trace:
W20240507 08:05:52.772176 10334 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 919) took 3571 ms (client timeout 1833 ms). Trace:
I20240507 08:05:52.904314 11625 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:52.907265 11625 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:52.906677 10334 rpcz_store.cc:269] 0507 08:05:49.200735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:50.772515 (+1571780us) service_pool.cc:225] Handling call
0507 08:05:52.771662 (+1999147us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b540013ea54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:52.771889 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":715000064}
W20240507 08:05:52.907676 10288 debug-util.cc:398] Leaking SignalData structure 0x7b080008b440 after lost signal to thread 10124
I20240507 08:05:52.908372 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.909200 10334 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
W20240507 08:05:52.780177 10340 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 923) took 3577 ms (client timeout 2206 ms). Trace:
W20240507 08:05:52.910710 10340 rpcz_store.cc:269] 0507 08:05:49.202660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.779956 (+3577296us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.780113 (+ 157us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.911851 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:52.912662 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 61) took 2243 ms. Trace:
I20240507 08:05:52.912974 10340 rpcz_store.cc:276] 0507 08:05:50.668953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.911490 (+2242537us) service_pool.cc:225] Handling call
0507 08:05:52.912604 (+ 1114us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.786450 10336 rpcz_store.cc:269] 0507 08:05:49.198571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:49.283729 (+ 85158us) service_pool.cc:225] Handling call
0507 08:05:51.178626 (+1894897us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54002c88d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:51.178784 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3465963904}
I20240507 08:05:52.787446 10939 rpcz_store.cc:276] 0507 08:05:48.321758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.405214 (+ 83456us) service_pool.cc:225] Handling call
0507 08:05:48.451950 (+ 46736us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:48.721950 (+270000us) raft_consensus.cc:1520] Early marking committed up to index 5
0507 08:05:49.124774 (+402824us) log.cc:842] Serialized 14 byte log entry
0507 08:05:49.138193 (+ 13419us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:49.138284 (+ 91us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:05:49.138346 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:51.227008 (+2088662us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b5400410354. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:51.227079 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:51.227314 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":399262,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":4068,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":1394,"spinlock_wait_cycles":578368640,"thread_start_us":341144,"threads_started":2,"wal-append.queue_time_us":12073,"wal-append.run_cpu_time_us":1057,"wal-append.run_wall_time_us":1001069}
W20240507 08:05:52.793942 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 929) took 2241 ms (client timeout 1577 ms). Trace:
W20240507 08:05:52.914288 10341 rpcz_store.cc:269] 0507 08:05:50.552761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.793770 (+2241009us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.793891 (+ 121us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.914424 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.915845 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 63) took 1850 ms. Trace:
W20240507 08:05:52.807586 10338 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 62) took 2119 ms (client timeout 1652 ms). Trace:
W20240507 08:05:52.917235 10338 rpcz_store.cc:269] 0507 08:05:50.688515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.807375 (+2118860us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.807499 (+ 124us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.917951 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.918581 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.919135 10340 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:52.919893 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 931) took 2290 ms. Trace:
I20240507 08:05:52.920305 10340 rpcz_store.cc:276] 0507 08:05:50.629037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.913563 (+2284526us) service_pool.cc:225] Handling call
0507 08:05:52.919797 (+ 6234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:52.922032 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 933) took 2247 ms. Trace:
I20240507 08:05:52.862108 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:50.882345 11619 raft_consensus.cc:3032] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:52.873384 10231 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:52.953611 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:52.954458 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:52.733412 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1251) took 1745 ms (client timeout 1583 ms). Trace:
I20240507 08:05:52.955749 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:52.955696 10191 rpcz_store.cc:269] 0507 08:05:50.988396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.733226 (+1744830us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.733351 (+ 125us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.956856 10936 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:52.957257 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.958393 10191 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:52.958837 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:52.959004 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:52.959846 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.960474 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:52.960876 10191 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:52.876914 11653 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:52.962157 11653 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:05:52.741554 10184 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1246) took 1754 ms (client timeout 1714 ms). Trace:
W20240507 08:05:52.963696 10184 rpcz_store.cc:269] 0507 08:05:50.986783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.741354 (+1754571us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.741487 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.965190 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.967020 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.857318 11712 raft_consensus.cc:3032] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:52.968478 11712 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:52.868670 11533 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:52.878355 10155 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:52.861496 11696 raft_consensus.cc:3037] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:52.880551 10156 leader_election.cc:334] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.577s (SENT)
W20240507 08:05:52.972690 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 76104.2us
I20240507 08:05:52.881618 11713 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:52.883634 11538 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:52.977013 11538 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:52.743866 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.977808 10189 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
W20240507 08:05:52.978335 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1235) took 2186 ms (client timeout 2062 ms). Trace:
W20240507 08:05:52.978699 10189 rpcz_store.cc:269] 0507 08:05:50.792282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.743198 (+1950916us) service_pool.cc:225] Handling call
0507 08:05:52.978264 (+235066us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.979521 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 106439us
I20240507 08:05:52.979737 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.897650 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.981348 10188 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:52.982095 10088 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59954 (request call id 159) took 7671 ms. Trace:
I20240507 08:05:52.982209 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:52.982323 10304 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 2.289s (SENT) [suppressed 64 similar messages]
I20240507 08:05:52.983145 10188 raft_consensus.cc:2396] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:52.984086 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.984830 10188 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:52.982493 10088 rpcz_store.cc:276] 0507 08:05:45.311198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:45.311377 (+ 179us) service_pool.cc:225] Handling call
0507 08:05:52.982046 (+7670669us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:52.836393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:52.836905 (+ 512us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:52.836954 (+ 49us) write_op.cc:435] Acquired schema lock
0507 08:05:52.836986 (+ 32us) tablet.cc:660] Decoding operations
0507 08:05:52.837419 (+ 433us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:52.837476 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:52.837513 (+ 37us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:05:52.837871 (+ 358us) tablet.cc:699] Row locks acquired
0507 08:05:52.837891 (+ 20us) write_op.cc:260] PREPARE: finished
0507 08:05:52.838044 (+ 153us) write_op.cc:270] Start()
0507 08:05:52.838191 (+ 147us) write_op.cc:276] Timestamp: P: 1715069152837990 usec, L: 0
0507 08:05:52.838224 (+ 33us) op_driver.cc:331] REPLICATION: starting
0507 08:05:52.838818 (+ 594us) log.cc:842] Serialized 928 byte log entry
0507 08:05:52.840866 (+ 2048us) op_driver.cc:444] REPLICATION: finished
0507 08:05:52.841342 (+ 476us) write_op.cc:301] APPLY: starting
0507 08:05:52.841417 (+ 75us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:52.841525 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:52.841636 (+ 111us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:52.842060 (+ 424us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:52.842229 (+ 169us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0507 08:05:52.842284 (+ 55us) write_op.cc:312] APPLY: finished
0507 08:05:52.842539 (+ 255us) log.cc:842] Serialized 28 byte log entry
0507 08:05:52.843344 (+ 805us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:52.843447 (+ 103us) write_op.cc:454] Released schema lock
0507 08:05:52.843504 (+ 57us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:52.843786 (+ 282us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:52.843815 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":257,"apply.run_cpu_time_us":2091,"apply.run_wall_time_us":2695,"num_ops":2,"prepare.queue_time_us":10677,"prepare.run_cpu_time_us":2707,"prepare.run_wall_time_us":2706,"replication_time_us":2552}]]}
W20240507 08:05:52.900322 10259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 904) took 4730 ms (client timeout 3118 ms). Trace:
W20240507 08:05:52.985718 10259 rpcz_store.cc:269] 0507 08:05:48.170333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:51.166538 (+2996205us) service_pool.cc:225] Handling call
0507 08:05:52.900124 (+1733586us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.777312 10332 rpcz_store.cc:269] 0507 08:05:49.199421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.776753 (+3577332us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.776884 (+ 131us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:52.986111 11712 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:52.987023 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.987785 11712 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:52.988529 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.989234 10332 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
W20240507 08:05:52.904310 10192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1244) took 1918 ms (client timeout 2014 ms). Trace:
W20240507 08:05:52.989696 10192 rpcz_store.cc:269] 0507 08:05:50.986282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.758336 (+1772054us) service_pool.cc:225] Handling call
0507 08:05:52.904226 (+145890us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.990279 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 112160us
I20240507 08:05:52.990625 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.999756 10192 raft_consensus.cc:2379] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.000675 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.906486 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.002076 10089 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59948 (request call id 163) took 5885 ms. Trace:
I20240507 08:05:53.002506 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.002445 10089 rpcz_store.cc:276] 0507 08:05:47.116670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.128712 (+ 12042us) service_pool.cc:225] Handling call
0507 08:05:53.001894 (+5873182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123570,"reported_tablets":11}
I20240507 08:05:53.003500 10190 raft_consensus.cc:2379] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.004381 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.005028 10190 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:52.906265 10187 rpcz_store.cc:269] 0507 08:05:47.247037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:47.391032 (+143995us) service_pool.cc:225] Handling call
0507 08:05:50.904051 (+3513019us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec572e 000000000033cd89 00007fcf95cf46c0 00007fcf95dba3c4 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:05:52.905662 (+2001611us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5293440}
I20240507 08:05:53.005235 10941 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:05:53.006199 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.007048 10187 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:53.008057 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.008736 10187 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:52.915725 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.010535 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.010812 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:53.011451 10341 raft_consensus.cc:2396] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:52.993284 11619 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.012234 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.012552 11732 tablet_replica.cc:330] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:05:53.012688 11619 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:51.073421 11617 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:53.001376 10192 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:52.923645 10339 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.016399 10907 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:52.924607 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 943) took 1834 ms (client timeout 1890 ms). Trace:
I20240507 08:05:53.017976 11713 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:52.925065 10334 rpcz_store.cc:276] 0507 08:05:50.674348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.907983 (+2233635us) service_pool.cc:225] Handling call
0507 08:05:52.921935 (+ 13952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.019469 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.020282 10334 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:52.904970 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 930) took 2285 ms. Trace:
I20240507 08:05:52.924759 10186 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:52.925750 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.021804 10337 rpcz_store.cc:276] 0507 08:05:50.619295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.903045 (+2283750us) service_pool.cc:225] Handling call
0507 08:05:52.904895 (+ 1850us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.023531 10340 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20240507 08:05:53.023633 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 908) took 3825 ms. Trace:
I20240507 08:05:53.024348 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 910) took 3826 ms. Trace:
I20240507 08:05:53.024099 10337 rpcz_store.cc:276] 0507 08:05:49.197771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.022591 (+3824820us) service_pool.cc:225] Handling call
0507 08:05:53.022998 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.023296 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.023324 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.023362 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:53.023396 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.023448 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.023546 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:52.925601 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
W20240507 08:05:53.025803 10186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1238) took 2233 ms (client timeout 2289 ms). Trace:
I20240507 08:05:52.928515 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:53.026186 10186 rpcz_store.cc:269] 0507 08:05:50.792762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.874965 (+2082203us) service_pool.cc:225] Handling call
0507 08:05:53.025732 (+150767us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.850411 10335 rpcz_store.cc:269] 0507 08:05:50.662032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.841153 (+2179121us) service_pool.cc:212] Skipping call since client already timed out
0507 08:05:52.841265 (+ 112us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20240507 08:05:53.027017 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.027712 10186 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.027892 10335 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:53.028534 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 915) took 3828 ms. Trace:
I20240507 08:05:53.028841 10335 rpcz_store.cc:276] 0507 08:05:49.199712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.027231 (+3827519us) service_pool.cc:225] Handling call
0507 08:05:53.027604 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.028323 (+ 719us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.028487 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:52.920364 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.025038 10340 rpcz_store.cc:276] 0507 08:05:49.198327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.022670 (+3824343us) service_pool.cc:225] Handling call
0507 08:05:53.023191 (+ 521us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:53.024027 (+ 836us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.024283 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.030089 10338 raft_consensus.cc:2379] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.031075 10340 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:05:53.031639 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 938) took 1965 ms. Trace:
I20240507 08:05:53.031714 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 64) took 1965 ms. Trace:
I20240507 08:05:53.032199 10340 rpcz_store.cc:276] 0507 08:05:51.065894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.030294 (+1964400us) service_pool.cc:225] Handling call
0507 08:05:53.030687 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.031466 (+ 779us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.031667 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.032090 10338 rpcz_store.cc:276] 0507 08:05:51.066482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.030684 (+1964202us) service_pool.cc:225] Handling call
0507 08:05:53.031040 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.031273 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.031303 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.031349 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:53.031399 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.031456 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.031564 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:52.924799 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:52.931051 11534 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:53.034392 11534 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.036442 11534 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:52.932438 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:52.938699 10185 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
W20240507 08:05:53.037086 10185 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1247) took 2050 ms (client timeout 2075 ms). Trace:
I20240507 08:05:52.943224 10183 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
W20240507 08:05:52.946131 11733 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.111s user 0.001s sys 0.000s
W20240507 08:05:53.037411 10185 rpcz_store.cc:269] 0507 08:05:50.987015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.866021 (+1879006us) service_pool.cc:225] Handling call
0507 08:05:53.037035 (+171014us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.038564 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.039378 10185 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:53.013345 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.042279 10341 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:53.043208 11713 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:05:52.950403 10306 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.045637 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:53.045949 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 95936.8us
I20240507 08:05:53.046274 11533 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:52.951988 11711 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:52.881008 11648 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:53.046598 10188 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.047616 11648 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:52.973376 10232 connection.cc:232] RPC call timeout handler was delayed by -4.2069712030000233s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20240507 08:05:53.049782 10232 connection.cc:232] RPC call timeout handler was delayed by -4.1815649210000174s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20240507 08:05:53.050123 10232 connection.cc:232] RPC call timeout handler was delayed by -4.0662173050000092s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20240507 08:05:53.050937 10232 connection.cc:232] RPC call timeout handler was delayed by -4.0285597029999849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20240507 08:05:53.051278 10232 connection.cc:232] RPC call timeout handler was delayed by -3.8568712029999688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2284s for a response
W20240507 08:05:52.972956 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 92506.4us
I20240507 08:05:53.045967 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 59) took 3911 ms. Trace:
I20240507 08:05:53.051919 10334 rpcz_store.cc:276] 0507 08:05:49.134075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.020988 (+3886913us) service_pool.cc:225] Handling call
0507 08:05:53.021405 (+ 417us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.045605 (+ 24200us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.045638 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.045677 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:05:53.045741 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.045790 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.045905 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:52.909442 10288 debug-util.cc:398] Leaking SignalData structure 0x7b0800321640 after lost signal to thread 10216
I20240507 08:05:52.991695 10155 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:52.991880 11579 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:52.992441 11601 consensus_queue.cc:1046] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:52.995914 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:52.997465 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:52.916466 10336 rpcz_store.cc:276] 0507 08:05:51.065565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.914004 (+1848439us) service_pool.cc:225] Handling call
0507 08:05:52.915756 (+ 1752us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.015028 11617 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:52.923085 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08003a5340 after lost signal to thread 10216
W20240507 08:05:53.017787 10333 rpcz_store.cc:269] 0507 08:05:51.090325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.861611 (+1771286us) service_pool.cc:225] Handling call
0507 08:05:52.924522 (+ 62911us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:52.927409 10934 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:53.071789 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.072991 11653 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.073242 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 65) took 1971 ms. Trace:
W20240507 08:05:53.037611 10183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1248) took 2050 ms (client timeout 2135 ms). Trace:
I20240507 08:05:53.074563 11617 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:53.074859 10183 rpcz_store.cc:269] 0507 08:05:50.987574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.886063 (+1898489us) service_pool.cc:225] Handling call
0507 08:05:53.037546 (+151483us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.074777 10334 rpcz_store.cc:276] 0507 08:05:51.101884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.062669 (+1960785us) service_pool.cc:225] Handling call
0507 08:05:53.072666 (+ 9997us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.072932 (+ 266us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.072965 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.073000 (+ 35us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:53.073025 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.073080 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.073180 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.075840 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.076761 10183 raft_consensus.cc:2379] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:52.945174 11625 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.046844 11533 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:52.974347 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:52.968065 10184 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.066143 11534 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:53.067757 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08003a5360 after lost signal to thread 10217
W20240507 08:05:53.060904 10288 debug-util.cc:398] Leaking SignalData structure 0x7b0800321620 after lost signal to thread 10217
I20240507 08:05:53.069178 10339 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:53.047122 11711 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.062866 11648 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.070363 11538 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:53.070910 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 913) took 3869 ms. Trace:
I20240507 08:05:53.071445 11619 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:52.941316 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:53.072748 10089 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 3 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:53.086551 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.079749 10185 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:53.072993 10306 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:53.083925 10906 leader_election.cc:334] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 2.197s (SENT)
I20240507 08:05:53.073657 11653 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.089829 10942 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20240507 08:05:53.087805 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:53.096815 10339 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 928) took 2606 ms (client timeout 2589 ms). Trace:
W20240507 08:05:53.156142 10232 connection.cc:232] RPC call timeout handler was delayed by -3.0706451760000277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
I20240507 08:05:53.165529 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 941) took 2098 ms. Trace:
I20240507 08:05:53.166427 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 942) took 2099 ms. Trace:
I20240507 08:05:53.158496 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.167570 10341 rpcz_store.cc:276] 0507 08:05:51.067112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.062892 (+1995780us) service_pool.cc:225] Handling call
0507 08:05:53.080018 (+ 17126us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.166111 (+ 86093us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.166136 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.166181 (+ 45us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:53.166209 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.166267 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.166373 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.167351 10338 rpcz_store.cc:276] 0507 08:05:51.066943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.063337 (+1996394us) service_pool.cc:225] Handling call
0507 08:05:53.090593 (+ 27256us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.165149 (+ 74556us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.165179 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.165218 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:53.165245 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.165312 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.165451 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:53.167007 10339 rpcz_store.cc:269] 0507 08:05:50.479450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:52.876135 (+2396685us) service_pool.cc:225] Handling call
0507 08:05:53.085991 (+209856us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.091176 10337 rpcz_store.cc:276] 0507 08:05:49.199139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.025421 (+3826282us) service_pool.cc:225] Handling call
0507 08:05:53.053425 (+ 28004us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.065008 (+ 11583us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.065036 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.066538 (+ 1502us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:53.066581 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.068609 (+ 2028us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5400333e54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:53.068657 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.068800 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26198912}
I20240507 08:05:53.149993 10192 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:53.171072 11579 consensus_queue.cc:1046] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:53.165483 10089 catalog_manager.cc:5484] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 3 to 4, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 1 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:53.171367 10908 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:53.172087 11579 raft_consensus.cc:3032] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:53.172466 11713 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:53.173257 11713 raft_consensus.cc:686] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:53.174602 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 940) took 2107 ms. Trace:
I20240507 08:05:53.172466 11579 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.174465 11713 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:53.149459 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 138851us
I20240507 08:05:53.172528 10089 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240507 08:05:53.169755 10232 connection.cc:232] RPC call timeout handler was delayed by -3.0292451760000176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20240507 08:05:53.176610 10232 connection.cc:232] RPC call timeout handler was delayed by -2.9401451760000441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20240507 08:05:53.177177 10232 connection.cc:232] RPC call timeout handler was delayed by -2.9176451760000077s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
I20240507 08:05:53.177559 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:53.177562 10232 connection.cc:232] RPC call timeout handler was delayed by -2.8960451760000296s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20240507 08:05:53.178659 10232 connection.cc:232] RPC call timeout handler was delayed by -2.7644737599999871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2929s for a response
W20240507 08:05:53.144297 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 58605.3us
W20240507 08:05:53.179059 10232 connection.cc:232] RPC call timeout handler was delayed by -2.5432451760000276s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
I20240507 08:05:53.085477 11736 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:53.180078 10232 connection.cc:232] RPC call timeout handler was delayed by -1.928545176s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2787s for a response
W20240507 08:05:53.180553 10232 connection.cc:232] RPC call timeout handler was delayed by -1.4956451760000391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3268s for a response
I20240507 08:05:53.180330 11736 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.181048 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 898) took 5161 ms. Trace:
I20240507 08:05:53.182826 11736 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.183514 11736 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:53.184271 11736 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.175024 10340 rpcz_store.cc:276] 0507 08:05:51.066806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.064807 (+1998001us) service_pool.cc:225] Handling call
0507 08:05:53.166634 (+101827us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.174261 (+ 7627us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.174292 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.174333 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:53.174368 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.174430 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.174553 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.181594 10261 rpcz_store.cc:276] 0507 08:05:48.019456 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:48.195003 (+175547us) service_pool.cc:225] Handling call
0507 08:05:48.211663 (+ 16660us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:48.394791 (+183128us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:48.394821 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:48.395657 (+ 836us) log.cc:842] Serialized 26 byte log entry
0507 08:05:48.433364 (+ 37707us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:52.897754 (+4464390us) log.cc:842] Serialized 14 byte log entry
0507 08:05:53.180029 (+282275us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.180597 (+ 568us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:53.180701 (+ 104us) raft_consensus.cc:1671] finished
0507 08:05:53.180720 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.180906 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10260,"prepare.queue_time_us":129029,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":49485,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":11071,"thread_start_us":4856270,"threads_started":5,"wal-append.queue_time_us":179557,"wal-append.run_cpu_time_us":24233,"wal-append.run_wall_time_us":1091447}
I20240507 08:05:53.088973 11625 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:53.191280 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 118420us
I20240507 08:05:53.191789 11617 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:53.185842 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.194548 10186 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:53.195117 11732 raft_consensus.cc:2229] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:53.090191 11538 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:53.198035 10156 leader_election.cc:334] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Remote error: Timed out: Call waited in the queue past client deadline
I20240507 08:05:53.199291 10190 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:05:53.090556 11711 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:53.201644 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 121254us
I20240507 08:05:53.201915 10191 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
W20240507 08:05:53.216703 10233 connection.cc:232] RPC call timeout handler was delayed by -3.4775653180000177s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20240507 08:05:53.217284 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1242) took 2421 ms. Trace:
I20240507 08:05:53.217658 10191 rpcz_store.cc:276] 0507 08:05:50.795881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.087500 (+2291619us) service_pool.cc:225] Handling call
0507 08:05:53.217232 (+129732us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.085475 11738 raft_consensus.cc:2741] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20240507 08:05:53.220659 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.224911 10158 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:53.215641 11538 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.222553 10089 catalog_manager.cc:5484] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 3 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:53.228503 10183 raft_consensus.cc:2379] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.089845 11534 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:53.230598 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 239127us
W20240507 08:05:53.231626 10288 debug-util.cc:398] Leaking SignalData structure 0x7b080023fa40 after lost signal to thread 10291
I20240507 08:05:53.231699 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.232864 11732 raft_consensus.cc:2258] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:53.235268 10187 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:53.202104 11579 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.236524 10336 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240507 08:05:53.236899 11617 consensus_queue.cc:1046] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:53.222107 10184 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.237746 11711 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:53.237460 11653 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:53.237766 10189 raft_consensus.cc:2396] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:53.240819 10188 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:53.087136 11533 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.247053 10186 raft_consensus.cc:2396] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240507 08:05:53.247970 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 66776.9us
W20240507 08:05:53.248358 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 186303us
I20240507 08:05:53.249158 10192 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.250679 10335 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240507 08:05:53.258016 10335 pending_rounds.cc:76] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 4
I20240507 08:05:53.257862 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1215) took 5121 ms. Trace:
W20240507 08:05:53.253327 10156 leader_election.cc:334] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): Remote error: Timed out: Call waited in the queue past client deadline
I20240507 08:05:53.252826 10336 pending_rounds.cc:76] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 4
W20240507 08:05:53.253327 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 51915.5us
I20240507 08:05:53.258477 10190 rpcz_store.cc:276] 0507 08:05:48.134864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.222531 (+5087667us) service_pool.cc:225] Handling call
0507 08:05:53.249924 (+ 27393us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.255698 (+ 5774us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.255723 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.255767 (+ 44us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:53.255797 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.255854 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.255958 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:53.249297 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08000aed60 after lost signal to thread 10291
I20240507 08:05:53.255597 10185 raft_consensus.cc:2379] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.230062 11740 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:53.227730 10908 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:53.262861 11736 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.262620 11711 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:53.259923 10213 debug-util.cc:398] Leaking SignalData structure 0x7b0800231740 after lost signal to thread 10292
I20240507 08:05:53.264415 11732 ts_tablet_manager.cc:1875] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:05:53.264855 11711 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:53.265275 10213 debug-util.cc:398] Leaking SignalData structure 0x7b08001bcc40 after lost signal to thread 10365
W20240507 08:05:53.265621 10213 debug-util.cc:398] Leaking SignalData structure 0x7b080037ca60 after lost signal to thread 10366
I20240507 08:05:53.265899 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.266624 10186 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20240507 08:05:53.267004 10213 debug-util.cc:398] Leaking SignalData structure 0x7b0800348cc0 after lost signal to thread 10967
I20240507 08:05:53.266574 11625 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:53.267375 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 52) took 4095 ms. Trace:
I20240507 08:05:53.267776 10185 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:53.266584 10259 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.268640 10192 raft_consensus.cc:1226] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:53.269167 10185 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:53.269487 10156 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.270364 11648 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:53.258903 10336 pending_rounds.cc:97] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240507 08:05:53.270752 11648 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:53.260860 10306 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.270862 10156 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.271203 11648 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.271562 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 47) took 5108 ms. Trace:
I20240507 08:05:53.258437 10335 pending_rounds.cc:97] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20240507 08:05:53.271939 10306 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:53.272433 10156 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.272841 10188 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:53.273208 10306 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:53.275892 11696 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.273758 11712 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:53.276729 11712 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:53.267396 10908 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:53.277122 11712 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.273973 11617 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:53.278496 11530 raft_consensus.cc:3032] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:53.278908 11530 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:53.277123 11742 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240507 08:05:53.280093 11530 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.273852 11625 raft_consensus.cc:686] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:53.282166 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.282532 10336 raft_consensus.cc:2866] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:53.283248 11625 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.284153 10185 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:53.272121 10335 raft_consensus.cc:2866] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:53.267673 11740 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.269189 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1222) took 4083 ms. Trace:
I20240507 08:05:53.284958 10192 rpcz_store.cc:276] 0507 08:05:49.185487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.261820 (+4076333us) service_pool.cc:225] Handling call
0507 08:05:53.268475 (+ 6655us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.269002 (+ 527us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.269147 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.285697 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 60) took 3609 ms. Trace:
I20240507 08:05:53.286031 10335 rpcz_store.cc:276] 0507 08:05:49.676339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.029538 (+3353199us) service_pool.cc:225] Handling call
0507 08:05:53.040606 (+ 11068us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.284930 (+244324us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.285491 (+ 561us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540013f454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:53.285651 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463249664}
I20240507 08:05:53.267928 10186 rpcz_store.cc:276] 0507 08:05:49.171522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.260823 (+4089301us) service_pool.cc:225] Handling call
0507 08:05:53.266310 (+ 5487us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.267150 (+ 840us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.267304 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.287376 10188 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240507 08:05:53.287878 10188 pending_rounds.cc:76] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 2
I20240507 08:05:53.288152 10188 pending_rounds.cc:97] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240507 08:05:53.288486 10188 raft_consensus.cc:2866] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:53.276479 10306 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.290489 11740 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:53.271941 10183 rpcz_store.cc:276] 0507 08:05:48.163465 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.247802 (+5084337us) service_pool.cc:225] Handling call
0507 08:05:53.271122 (+ 23320us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.271286 (+ 164us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.271309 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.271350 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:05:53.271379 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.271425 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.271515 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:53.291874 10288 debug-util.cc:398] Leaking SignalData structure 0x7b08003733e0 after lost signal to thread 10365
I20240507 08:05:53.270882 10189 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
W20240507 08:05:53.293267 10288 debug-util.cc:398] Leaking SignalData structure 0x7b0800330340 after lost signal to thread 10967
I20240507 08:05:53.294016 11579 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.282166 11653 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:53.296020 11713 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:05:53.296998 11617 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.297565 11617 raft_consensus.cc:686] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:53.288213 10186 raft_consensus.cc:1261] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:53.298525 11617 consensus_queue.cc:237] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.254323 11738 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:53.301430 11738 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.284181 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 66) took 2155 ms. Trace:
I20240507 08:05:53.301903 11738 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:53.267324 10213 debug-util.cc:398] Leaking SignalData structure 0x7b0800348c80 after lost signal to thread 10968
I20240507 08:05:53.302045 10306 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.302022 10336 rpcz_store.cc:276] 0507 08:05:51.129105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.063753 (+1934648us) service_pool.cc:225] Handling call
0507 08:05:53.066109 (+ 2356us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.283029 (+216920us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.283951 (+ 922us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400069d54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:53.284109 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":391233920}
I20240507 08:05:53.303951 10158 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.306154 10156 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.307363 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.309871 10306 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.303089 11695 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.313910 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.303925 11530 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.317857 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.318481 11654 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:53.318859 11654 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.319092 11695 raft_consensus.cc:686] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:53.319022 11623 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:53.291230 11534 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:53.307102 11619 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:05:53.315323 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52891.6us
I20240507 08:05:53.327644 11699 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:53.328428 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.329173 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.327718 10188 raft_consensus.cc:1261] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:53.315023 11713 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.367687 11625 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:05:53.306639 11648 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.326335 11619 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:53.371049 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 62353.5us
I20240507 08:05:53.334393 11695 consensus_queue.cc:237] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:53.309000 11739 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:05:53.408330 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 104504us
I20240507 08:05:53.325771 11538 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.408185 10187 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:53.411955 10259 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.414076 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:53.415045 10264 raft_consensus.cc:2379] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.415927 10190 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:53.422852 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1214) took 5292 ms. Trace:
I20240507 08:05:53.423219 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1225) took 4153 ms. Trace:
I20240507 08:05:53.424723 10190 rpcz_store.cc:276] 0507 08:05:49.267342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.261562 (+3994220us) service_pool.cc:225] Handling call
0507 08:05:53.367031 (+105469us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.420132 (+ 53101us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.420333 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:53.412964 11619 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.423322 10191 rpcz_store.cc:276] 0507 08:05:48.125510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.221085 (+5095575us) service_pool.cc:225] Handling call
0507 08:05:53.260232 (+ 39147us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.418043 (+157811us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:53.418076 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:53.418113 (+ 37us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:53.418138 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.418195 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:53.418292 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29928,"raft.queue_time_us":91261,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":258,"thread_start_us":110015,"threads_started":1}
I20240507 08:05:53.414819 11739 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:53.417398 11740 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.413727 11713 raft_consensus.cc:686] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:53.435086 10258 raft_consensus.cc:2379] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.437860 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 48) took 5269 ms. Trace:
I20240507 08:05:53.441776 10187 rpcz_store.cc:276] 0507 08:05:48.163771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.308422 (+5144651us) service_pool.cc:225] Handling call
0507 08:05:53.331236 (+ 22814us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:53.422286 (+ 91050us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:53.433500 (+ 11214us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:53.444123 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 78476.2us
W20240507 08:05:53.416213 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 96719.6us
I20240507 08:05:53.451622 10232 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.455036 11657 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:53.455571 11657 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:53.455979 11657 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:53.456293 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.457736 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.458084 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.458413 10264 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.458814 10260 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:53.459352 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.459615 11696 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:53.459270 11712 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.460129 10264 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
W20240507 08:05:53.462381 11747 log.cc:924] Time spent T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.053s user 0.003s sys 0.000s
I20240507 08:05:53.463272 11601 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:53.463752 11601 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:53.464118 11601 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.464558 10184 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:05:53.450706 11740 raft_consensus.cc:686] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:53.466209 11648 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.466502 11740 consensus_queue.cc:237] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.437812 11739 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:53.470916 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 105245us
W20240507 08:05:53.474548 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44815.1us
W20240507 08:05:53.475353 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 185629us
I20240507 08:05:53.478052 10262 raft_consensus.cc:1226] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:53.478294 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.479190 10266 raft_consensus.cc:2379] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.458480 11538 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.480361 11745 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.481176 11749 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:53.482476 10185 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:53.461449 11712 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:53.490032 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.490669 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:53.493772 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 188947us
I20240507 08:05:53.495829 11699 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:53.478236 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.498167 10263 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.498484 11699 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.501670 10258 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:53.502329 10266 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:53.496357 11745 raft_consensus.cc:686] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:53.511209 11745 consensus_queue.cc:237] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.513988 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.513741 11657 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.518278 11743 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:53.518709 11743 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.519233 11743 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:53.447642 11654 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.515030 10156 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:53.509316 11742 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:53.533870 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.534634 10266 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.543051 10258 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:53.545595 10231 leader_election.cc:398] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:53.545970 10231 leader_election.cc:401] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240507 08:05:53.545989 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.546862 10263 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.533473 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.550240 10265 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.551013 11695 consensus_queue.cc:1046] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:53.552111 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.552983 10266 raft_consensus.cc:2379] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.552682 10942 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.553354 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.554569 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.554869 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.526436 11657 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:53.560705 10267 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.572170 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.573168 10263 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:53.573249 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.573617 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.574067 10337 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.574867 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.575409 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.576586 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.577903 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.578609 10262 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.578687 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.579047 10267 raft_consensus.cc:1226] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:53.579499 10334 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.579829 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.580413 10259 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
W20240507 08:05:53.580598 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 48519.2us
I20240507 08:05:53.580442 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.581843 10334 raft_consensus.cc:2354] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
I20240507 08:05:53.581662 11712 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:53.579509 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.582790 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.583005 11623 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:05:53.583217 11538 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:53.583619 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.583962 11623 raft_consensus.cc:3032] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:53.584378 11623 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:53.584812 10232 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.585095 11538 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:53.585700 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.586318 10263 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.586733 11736 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:53.587431 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.587672 11754 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:53.588053 10259 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.588137 11754 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:53.588647 11754 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:53.537214 11713 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.594597 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.598397 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.598919 10267 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.599228 10332 raft_consensus.cc:2396] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:53.599308 10339 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:53.600428 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.589520 10156 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.594349 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.601994 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.602022 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.603047 10266 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.604598 11753 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:53.604945 11654 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.605162 11753 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.573269 11732 ts_tablet_manager.cc:1888] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:53.606149 11732 log.cc:1196] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/e0f47233c3be4fe99e097ade2aaec1e8
I20240507 08:05:53.606174 11753 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.609910 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.610738 10266 raft_consensus.cc:2379] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.611512 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.612362 11738 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.612903 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.613459 10192 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:53.613936 10259 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.597158 11737 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3.
I20240507 08:05:53.531924 11742 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:53.597910 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 90667.7us
I20240507 08:05:53.571498 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.619027 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.620865 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.625037 10258 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.615135 11739 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.627233 10263 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:53.630295 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.630349 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.633811 10341 raft_consensus.cc:2396] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:53.633637 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.634420 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.635630 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.635852 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.636176 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.636796 10334 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.636376 10332 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.618644 11742 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:53.637955 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 78599.6us
I20240507 08:05:53.639295 11696 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:53.640800 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:53.641726 10906 leader_election.cc:334] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.767s (SENT)
I20240507 08:05:53.644079 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:53.644258 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.645000 10232 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.645233 10341 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.648247 11653 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:53.648660 11653 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.649102 11653 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:53.649107 10232 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.650071 11740 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:53.650194 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.650490 11740 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.650935 11740 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:53.651062 11711 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:53.623889 11625 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.021s
I20240507 08:05:53.596710 11601 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.654985 11739 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.651551 11711 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.655936 11711 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:53.655838 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.656495 11601 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:53.590332 11749 consensus_queue.cc:1046] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20240507 08:05:53.658362 11749 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:53.658480 10232 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.658725 11749 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.659891 10232 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.659571 11749 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.662482 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.585214 11623 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:53.671991 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.632184 10908 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:05:53.674567 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.674856 10334 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:53.679587 10232 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.687237 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:53.687947 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 54627us
I20240507 08:05:53.662302 11579 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:53.690152 10156 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.690639 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.691808 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.692229 10334 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.671284 11538 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:53.693646 10340 raft_consensus.cc:2396] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20240507 08:05:53.698155 11538 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:53.694082 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:53.694259 11625 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:53.694365 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet e0f47233c3be4fe99e097ade2aaec1e8 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:53.682953 11743 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:53.699966 10075 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 81406.3us
I20240507 08:05:53.697943 11619 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.700341 10341 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:53.674582 11737 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:53.705873 11538 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.699839 11530 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20240507 08:05:53.730084 11760 log.cc:924] Time spent T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20240507 08:05:53.731503 11749 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.718636 11696 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:53.779397 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.781721 11579 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.781793 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.783952 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:53.787734 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 154159us
I20240507 08:05:53.814919 10338 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:53.846648 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.872591 10341 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:53.872509 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:53.873303 10940 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:53.874230 10261 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:53.874425 10185 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:53.874599 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.875226 10188 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:53.875845 10339 raft_consensus.cc:2379] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.876304 11756 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:53.689886 11695 consensus_queue.cc:1035] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
W20240507 08:05:53.695286 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 76516.4us
I20240507 08:05:53.786850 10232 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:53.864516 10332 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:53.864081 11533 raft_consensus.cc:3032] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:53.867678 10334 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:53.786041 11625 raft_consensus.cc:686] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:53.884825 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.885471 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.886268 10190 raft_consensus.cc:2379] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.887708 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:53.714437 9051 debug-util.cc:398] Leaking SignalData structure 0x7b080000ed80 after lost signal to thread 10231
I20240507 08:05:53.872094 10337 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.876626 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.878103 10156 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.888844 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:53.889521 10334 raft_consensus.cc:2379] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.889880 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.889971 10156 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:53.884682 10265 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.890013 11654 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:53.890594 10339 raft_consensus.cc:2379] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:53.890795 11654 raft_consensus.cc:686] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.891152 11739 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:53.891661 11739 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.891762 11654 consensus_queue.cc:237] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.878258 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.893620 10192 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:53.885474 10338 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:05:53.897187 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.897737 10188 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:53.898169 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:53.897322 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 117921us
I20240507 08:05:53.905520 10261 raft_consensus.cc:1261] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:53.870134 11738 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:53.906880 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:53.910681 10191 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.914247 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:53.915261 10185 raft_consensus.cc:2379] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:53.882136 11533 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:05:53.921247 10906 leader_election.cc:334] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 2.145s (SENT)
I20240507 08:05:53.869688 10262 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:05:53.892534 11739 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:53.928532 10189 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:53.929761 10262 raft_consensus.cc:595] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:05:53.930434 10262 raft_consensus.cc:603] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:05:53.899266 11579 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:53.952744 10155 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:53.917171 11534 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
W20240507 08:05:53.962421 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 50630.9us
I20240507 08:05:53.963816 10261 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240507 08:05:53.964978 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 86828us
I20240507 08:05:53.954269 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.968542 10332 raft_consensus.cc:1226] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:53.985384 11751 raft_consensus.cc:3032] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Stepping down as leader of term 4
W20240507 08:05:53.985718 11763 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20240507 08:05:53.985998 11751 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:53.986514 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.987023 11744 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:53.987185 10265 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.987766 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:53.990159 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.991915 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.992484 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.966080 11753 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:53.993131 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.993935 10264 raft_consensus.cc:2362] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3. Re-sending same reply.
I20240507 08:05:53.994298 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:53.993963 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.994830 10258 raft_consensus.cc:2362] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3. Re-sending same reply.
I20240507 08:05:53.995185 10259 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:53.989367 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:53.986985 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 68985.7us
I20240507 08:05:53.997174 10231 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:53.951426 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.008095 10189 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:53.985514 11579 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.011313 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.986886 11751 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.011848 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.012307 11751 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:54.012533 10265 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:54.013957 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.015079 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.016234 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.016399 11653 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.017300 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.017869 11619 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.011947 10260 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:53.993353 10266 raft_consensus.cc:2379] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:53.955016 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.018280 11625 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.037122 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:53.968809 11743 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.043383 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:54.044109 10336 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.045006 10155 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:53.995985 11534 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.043174 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:54.047757 10189 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:05:54.048843 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.051307 10334 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:54.052563 10337 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:05:54.050110 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.061213 11773 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:54.063721 10940 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:54.064321 11743 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240507 08:05:54.065124 11764 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20240507 08:05:54.043062 11737 raft_consensus.cc:686] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.066939 10341 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:54.069425 10338 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
W20240507 08:05:54.074111 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 85740.6us
I20240507 08:05:54.037854 11711 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.078203 10158 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.085428 11741 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:54.086120 11741 raft_consensus.cc:686] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:54.087152 11741 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.066074 10187 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.066443 11738 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:54.052428 11740 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:54.108971 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 141194us
I20240507 08:05:54.037624 11533 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.111290 10231 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:54.112200 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 75395.3us
I20240507 08:05:54.119684 10336 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:54.120817 10156 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:54.107569 11653 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.067849 11742 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.129716 11777 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20240507 08:05:54.125941 11737 consensus_queue.cc:237] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.129889 10333 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:54.131124 11711 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.182763 10341 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:54.133498 11740 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.083506 11745 consensus_queue.cc:1046] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
W20240507 08:05:54.117349 11754 consensus_meta.cc:287] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.528s user 0.005s sys 0.006s
I20240507 08:05:54.187711 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:54.126073 11657 raft_consensus.cc:3032] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:54.141187 10231 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.142076 11783 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.191358 11657 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:54.137241 11738 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:54.145937 11776 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240507 08:05:54.141352 11761 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.270s user 0.001s sys 0.000s
I20240507 08:05:54.147766 10155 leader_election.cc:398] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:54.148223 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:54.181968 11760 log.cc:924] Time spent T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20240507 08:05:54.146672 11773 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:05:54.188108 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.190770 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:54.191069 11781 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20240507 08:05:54.192934 10189 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:54.149202 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:54.192268 11623 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:54.120991 11743 consensus_queue.cc:1046] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20240507 08:05:54.199621 10188 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:54.117043 11534 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.218662 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:54.221586 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:54.222203 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 105356us
I20240507 08:05:54.201021 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.196732 11738 raft_consensus.cc:686] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:54.225137 10189 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:54.227187 10188 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:54.228368 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:54.229076 10192 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:54.230744 10185 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:54.233709 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:54.235455 10304 leader_election.cc:334] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.518s (SENT)
I20240507 08:05:54.235841 10304 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.237409 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.240726 10155 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.195382 11754 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.241611 11753 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:54.242202 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.243299 11754 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:54.192297 11657 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.243484 11741 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.244230 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.246775 10190 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240507 08:05:54.248531 10189 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:05:54.250032 10232 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:54.253731 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.254876 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:54.256531 10185 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:05:54.256526 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:54.220413 11601 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20240507 08:05:54.243528 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.262001 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:54.262408 11738 consensus_queue.cc:237] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.262950 10184 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:54.263779 11742 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.264261 10184 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:54.264950 11778 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:05:54.264209 10304 leader_election.cc:334] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.580s (SENT)
I20240507 08:05:54.262912 10940 raft_consensus.cc:2396] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240507 08:05:54.354983 10304 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.259204 10185 raft_consensus.cc:1261] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:54.267290 11713 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:05:54.257668 11740 raft_consensus.cc:3032] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:05:54.279569 11653 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.224063 11530 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20240507 08:05:54.219316 11775 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20240507 08:05:54.266244 11696 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240507 08:05:54.352110 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 111504us
I20240507 08:05:54.265460 11778 raft_consensus.cc:686] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:54.355343 10306 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:54.362143 11791 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20240507 08:05:54.352494 11745 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:54.372752 10185 raft_consensus.cc:1226] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:54.369772 10192 raft_consensus.cc:1226] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:54.353677 11785 raft_consensus.cc:2741] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240507 08:05:54.371004 10158 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.369073 10942 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:54.372323 11745 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:54.358134 11740 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.380754 11653 consensus_queue.cc:1046] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240507 08:05:54.382164 11654 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:05:54.384496 11653 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:05:54.384932 11653 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.386032 11653 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.387595 11653 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:54.357874 11773 consensus_queue.cc:1046] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20240507 08:05:54.402163 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 49511.2us
W20240507 08:05:54.388621 11538 consensus_meta.cc:287] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.514s user 0.009s sys 0.000s
I20240507 08:05:54.357925 11737 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20240507 08:05:54.382615 11745 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:54.408473 10306 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:54.373651 10187 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
W20240507 08:05:54.407653 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 143547us
W20240507 08:05:54.410916 11789 log.cc:924] Time spent T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.131s user 0.000s sys 0.031s
I20240507 08:05:54.430121 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.430675 10340 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:54.398180 11699 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.408737 11773 consensus_queue.cc:1225] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240507 08:05:54.433239 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.433631 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.433495 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.434351 10262 raft_consensus.cc:2379] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:54.434471 10265 raft_consensus.cc:2362] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4. Re-sending same reply.
I20240507 08:05:54.406308 11538 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.436533 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:54.437165 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.438117 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:54.417063 11793 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20240507 08:05:54.400907 11785 consensus_queue.cc:1046] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20240507 08:05:54.439453 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.440184 11785 raft_consensus.cc:3032] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:54.440631 11785 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.433876 10260 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:54.440269 10089 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 2 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:54.441434 11785 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.427942 11530 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:54.443251 11530 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:05:54.428082 10304 leader_election.cc:334] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.583s (SENT)
I20240507 08:05:54.438011 11601 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:54.417543 11787 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:54.445428 10089 catalog_manager.cc:5484] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 4, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 4 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:54.448951 10089 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 2 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 3 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:05:54.460290 11745 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.450944 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.451553 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.452344 10186 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240507 08:05:54.453394 11787 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:54.438062 11538 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.442569 11785 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:54.455516 11737 consensus_queue.cc:1046] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:54.420539 11778 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:54.352126 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 211100us
W20240507 08:05:54.460137 10304 leader_election.cc:334] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.624s (SENT)
I20240507 08:05:54.473887 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.484838 10231 leader_election.cc:398] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
I20240507 08:05:54.489320 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.484885 10259 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:54.470453 11787 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:54.494391 10155 connection.cc:232] RPC call timeout handler was delayed by -3.4464598170000045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3118s for a response
I20240507 08:05:54.460839 11530 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.482434 11791 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:05:54.508297 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.509162 10259 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 1.
I20240507 08:05:54.458628 11740 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.523501 11740 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:54.524665 11711 raft_consensus.cc:3032] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:54.525034 11711 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.498484 11737 consensus_queue.cc:1225] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 4.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20240507 08:05:54.526371 11538 raft_consensus.cc:3032] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:54.526752 11538 raft_consensus.cc:729] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.527504 11538 consensus_queue.cc:260] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.519867 11799 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:54.525892 11711 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.529243 10231 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.521418 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.530503 10189 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:54.537134 10265 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:54.535697 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.537990 10262 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
W20240507 08:05:54.539562 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44595.4us
I20240507 08:05:54.504941 11617 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.496984 11775 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20240507 08:05:54.541052 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.498298 11695 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240507 08:05:54.558036 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.558892 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.546047 10259 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:54.558971 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.559532 11796 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:54.559906 10265 raft_consensus.cc:2379] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:54.560019 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.560868 10259 raft_consensus.cc:2379] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:54.560798 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.546582 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.561672 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.514360 11744 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.562306 10262 raft_consensus.cc:2379] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:54.562474 10259 raft_consensus.cc:2396] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:54.562709 11744 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.545722 11538 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.559070 10258 raft_consensus.cc:2396] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240507 08:05:54.545354 11534 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.553535 11653 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240507 08:05:54.566478 11653 raft_consensus.cc:3032] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:54.566895 11653 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.494122 11699 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20240507 08:05:54.567703 11653 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.558015 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.569233 11653 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.569262 10264 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:54.502136 11791 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240507 08:05:54.563145 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.553212 10306 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:54.559965 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.573024 10267 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:54.602566 10264 raft_consensus.cc:1261] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:54.560017 11796 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:54.606789 10262 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:54.607800 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.536577 11799 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.613560 10267 raft_consensus.cc:1226] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:54.610356 10183 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:54.610915 10262 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:05:54.617259 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 88134us
I20240507 08:05:54.619707 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.620278 10262 raft_consensus.cc:1261] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:54.621170 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.621973 10258 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
W20240507 08:05:54.602558 11800 log.cc:924] Time spent T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240507 08:05:54.625916 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.628463 10231 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.628629 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.626714 10264 raft_consensus.cc:2362] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3. Re-sending same reply.
I20240507 08:05:54.628553 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.633845 10337 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:54.629478 10259 raft_consensus.cc:2362] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5. Re-sending same reply.
I20240507 08:05:54.638458 10186 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240507 08:05:54.606238 11617 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240507 08:05:54.618691 11796 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:54.643179 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 102262us
I20240507 08:05:54.633968 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:54.646878 10942 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:54.624472 11791 consensus_queue.cc:1225] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240507 08:05:54.652325 10158 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:54.654057 10231 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.654354 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.673738 11810 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:54.674551 11799 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.654898 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:54.675662 11799 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.680240 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:54.681092 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 221065us
I20240507 08:05:54.684552 10187 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
W20240507 08:05:54.681929 10304 leader_election.cc:334] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.633s (SENT)
I20240507 08:05:54.650038 11711 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.695003 10232 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:54.672154 11745 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:54.700043 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.705834 10337 raft_consensus.cc:1261] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240507 08:05:54.683954 11749 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:05:54.713971 10304 leader_election.cc:334] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.663s (SENT)
I20240507 08:05:54.714363 10304 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.697237 11534 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:54.722947 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:54.709867 11810 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:54.712929 10335 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:54.711201 11749 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:54.713511 11617 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:05:54.772893 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56299.5us
I20240507 08:05:54.774931 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.777534 10259 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:54.761962 11749 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.722802 11579 consensus_queue.cc:1035] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20240507 08:05:54.800747 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:54.802167 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 88266us
I20240507 08:05:54.778883 11744 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:05:54.810408 10304 leader_election.cc:334] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.666s (SENT)
I20240507 08:05:54.810799 10304 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:54.777633 11810 raft_consensus.cc:686] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:54.783121 11619 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:54.820580 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45800.7us
I20240507 08:05:54.834523 10942 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:54.836799 10231 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:54.838280 10231 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.812731 11744 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:54.818277 11538 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.814231 11712 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.863062 10264 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240507 08:05:54.865725 10231 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:54.867811 11819 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20240507 08:05:54.870457 11740 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.870544 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:54.871179 11821 log.cc:924] Time spent T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20240507 08:05:54.873242 11534 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:54.873878 11534 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:54.874850 10231 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:54.875429 10304 leader_election.cc:334] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.668s (SENT)
I20240507 08:05:54.863442 11785 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:54.825718 11619 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:54.887120 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:54.863615 11773 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:54.858829 11744 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:54.913318 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.930119 10265 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:54.930395 10186 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:54.932276 10185 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:05:54.943675 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56545.3us
I20240507 08:05:54.875469 11579 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:54.912448 11538 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:54.946893 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:54.949162 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:54.933830 11785 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:54.935411 11773 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:54.953117 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 51041.8us
I20240507 08:05:54.964854 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:54.965991 10304 leader_election.cc:334] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.699s (SENT)
I20240507 08:05:54.966332 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:54.967850 10260 raft_consensus.cc:2396] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240507 08:05:54.946784 11617 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:54.955633 11745 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:54.959080 11773 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:54.873878 11810 consensus_queue.cc:237] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:54.974504 10304 leader_election.cc:334] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.714s (SENT)
W20240507 08:05:54.972972 11805 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.236s user 0.000s sys 0.001s
W20240507 08:05:54.975315 11815 log.cc:924] Time spent T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
W20240507 08:05:54.975394 11812 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.167s user 0.001s sys 0.000s
I20240507 08:05:54.975787 11831 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:05:54.978639 11823 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.115s user 0.000s sys 0.000s
I20240507 08:05:54.978685 11619 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:55.014503 11817 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20240507 08:05:55.014918 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:54.955265 11740 consensus_queue.cc:1046] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240507 08:05:55.017843 10264 raft_consensus.cc:1261] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240507 08:05:54.973361 11742 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:55.030396 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 190712us
I20240507 08:05:54.957118 11787 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.027326 10341 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:55.034236 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:55.033718 10906 leader_election.cc:334] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.885s (SENT)
I20240507 08:05:54.956705 11579 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:55.041652 10304 leader_election.cc:334] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.734s (SENT)
I20240507 08:05:55.044714 11814 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.045195 11814 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:55.046030 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.047153 11814 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:55.050426 11617 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.051285 11777 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:05:54.684036 11743 consensus_queue.cc:1159] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "59382e6647c549d89d83c70233c0667e" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]" } } last_received_current_leader { term: 0 index: 0 } }
W20240507 08:05:55.056291 11798 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240507 08:05:55.057937 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.059273 11811 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:55.059813 11811 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.038232 11657 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:55.073129 10341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:05:55.043583 11736 raft_consensus.cc:3032] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:55.101426 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.105732 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.105671 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.107775 10267 raft_consensus.cc:3032] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:55.108355 10341 raft_consensus.cc:595] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:05:55.109951 10341 raft_consensus.cc:603] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240507 08:05:55.113103 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 74337us
I20240507 08:05:55.108240 10267 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.115970 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:05:55.100940 11847 log.cc:924] Time spent T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20240507 08:05:55.040679 11742 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.132850 10260 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240507 08:05:55.143031 10304 leader_election.cc:334] T e0f47233c3be4fe99e097ade2aaec1e8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.746s (SENT)
I20240507 08:05:55.105530 11736 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.155058 10190 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:55.154507 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:55.156615 10189 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 4
I20240507 08:05:55.111862 11745 raft_consensus.cc:686] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.166182 10263 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:05:55.119652 11787 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:55.128780 11533 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:55.131244 10267 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:55.173835 10260 raft_consensus.cc:3032] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:55.177541 10260 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.181643 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:55.182540 10190 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:55.185676 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
W20240507 08:05:55.195327 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 179052us
I20240507 08:05:55.201928 10156 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:55.215647 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 72708.8us
W20240507 08:05:55.216599 10304 leader_election.cc:334] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.750s (SENT)
W20240507 08:05:55.217450 10304 leader_election.cc:334] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.753s (SENT)
I20240507 08:05:55.228001 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:55.230968 10341 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:55.234510 10338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:05:55.235126 10338 raft_consensus.cc:595] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:05:55.235463 10338 raft_consensus.cc:603] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:05:55.184554 11750 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
W20240507 08:05:55.242584 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 46683.5us
W20240507 08:05:55.247123 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 124709us
W20240507 08:05:55.237530 11653 consensus_meta.cc:287] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.668s user 0.011s sys 0.000s
I20240507 08:05:55.165745 11832 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20240507 08:05:55.251396 10184 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20240507 08:05:55.252782 11838 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20240507 08:05:55.261705 11849 consensus_queue.cc:1046] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:55.262203 11811 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:55.263213 11799 consensus_meta.cc:287] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.587s user 0.011s sys 0.001s
I20240507 08:05:55.263608 11617 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:55.264667 10233 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:55.260263 10189 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:55.268918 10306 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:55.272745 11813 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:55.273109 10184 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:55.273252 11813 raft_consensus.cc:686] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:55.273514 11579 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:55.274432 10233 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:55.260365 11835 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:55.276381 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:55.295336 10335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:05:55.263228 11849 raft_consensus.cc:3032] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:55.274128 11851 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:55.296073 10335 raft_consensus.cc:595] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:05:55.300520 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:55.305292 10335 raft_consensus.cc:603] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:05:55.274235 11813 consensus_queue.cc:237] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.277132 11833 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:55.276417 11811 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:55.286927 11835 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:55.261215 11745 consensus_queue.cc:237] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.296530 11653 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:55.299561 11849 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:55.267474 11818 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
I20240507 08:05:55.236884 11736 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:55.351878 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 47586.2us
I20240507 08:05:55.355580 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:55.324904 11833 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:55.305485 11846 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20240507 08:05:55.339560 11835 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:55.337862 11742 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.372702 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:55.374470 10156 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:55.299922 11825 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20240507 08:05:55.362288 11833 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:55.340595 11859 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.232s
W20240507 08:05:55.411648 11857 log.cc:924] Time spent T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20240507 08:05:55.420136 10190 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:55.421023 10334 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:05:55.421785 10334 raft_consensus.cc:595] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:05:55.424212 11849 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.425539 11849 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:55.427184 11744 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.429165 11744 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.328125 11773 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.431440 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:55.373987 11799 consensus_queue.cc:1159] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5d6656a616b747d393ff3de5328d2540" responder_term: 2 status { last_received { term: 2 index: 4 } last_committed_idx: 3 last_received_current_leader { term: 2 index: 4 } }
I20240507 08:05:55.434602 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:55.435271 10942 raft_consensus.cc:1799] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Handling vote request from an unknown peer bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:55.438591 10942 raft_consensus.cc:2379] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
W20240507 08:05:55.429723 10266 consensus_meta.cc:287] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.526s user 0.007s sys 0.003s
I20240507 08:05:55.439335 10266 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
W20240507 08:05:55.439958 11858 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
W20240507 08:05:55.441510 11856 log.cc:924] Time spent T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
W20240507 08:05:55.442610 11863 log.cc:924] Time spent T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20240507 08:05:55.415040 11579 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:55.312508 11851 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.459715 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.460739 10264 raft_consensus.cc:2379] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:55.461313 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:55.464013 10185 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:55.420192 11811 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20240507 08:05:55.470029 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:55.466804 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.471408 10259 raft_consensus.cc:2354] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:55.469436 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240507 08:05:55.481398 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:55.487303 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 269983us
I20240507 08:05:55.445756 11742 consensus_queue.cc:1046] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20240507 08:05:55.469584 10340 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:55.495079 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.495839 10264 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:05:55.484822 11835 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:55.505841 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 104599us
W20240507 08:05:55.513870 10304 leader_election.cc:334] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.771s (SENT)
I20240507 08:05:55.519555 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:55.524335 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 119222us
I20240507 08:05:55.519253 10940 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240507 08:05:55.525975 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.526533 10264 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:55.530236 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:05:55.531100 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:55.533088 11625 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.533615 11852 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20240507 08:05:55.533733 11625 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:55.534250 11749 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.725s user 0.009s sys 0.002s
I20240507 08:05:55.534686 11749 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:55.535887 10267 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:55.533175 11696 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
W20240507 08:05:55.489260 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 234244us
I20240507 08:05:55.537340 11749 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:55.541989 11696 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:55.552939 11849 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.567018 11849 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.567277 11696 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.568534 11696 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:55.568373 10260 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:55.569113 11849 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.569790 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:55.556686 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet e9adf1535c974b2abf910f3c1c537627 with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42910
I20240507 08:05:55.557839 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:55.574491 10335 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:55.557996 11773 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.569483 10260 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:55.571992 10155 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:55.572705 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:55.577334 10339 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:55.572793 10267 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240507 08:05:55.577627 10233 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:55.581230 10304 connection.cc:232] RPC call timeout handler was delayed by -1.0718128360000492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20240507 08:05:55.586524 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:55.586776 10304 connection.cc:232] RPC call timeout handler was delayed by -1.0624714820000349s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20430000000000004s for a response
W20240507 08:05:55.587175 10304 connection.cc:232] RPC call timeout handler was delayed by -1.0309687650000114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
W20240507 08:05:55.587546 10304 connection.cc:232] RPC call timeout handler was delayed by -1.0234454170000049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
I20240507 08:05:55.588506 10233 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:55.585816 10156 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:55.598784 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.596554 10335 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:55.600937 10155 leader_election.cc:398] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:55.601285 10155 leader_election.cc:401] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240507 08:05:55.586274 11861 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:55.592914 11744 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:55.593047 11773 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.593636 11851 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.605727 11776 raft_consensus.cc:3032] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:55.596998 11742 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:55.619963 10264 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:55.632194 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.633468 10264 raft_consensus.cc:2396] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240507 08:05:55.635465 11878 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240507 08:05:55.623760 11744 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.624192 11776 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.602638 11855 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20240507 08:05:55.640740 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.642359 10264 raft_consensus.cc:2379] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:55.645777 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.639863 11787 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.651997 11776 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.620703 11773 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.652287 11787 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.656374 11538 raft_consensus.cc:3032] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:55.628960 11751 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20240507 08:05:55.660900 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.663800 10264 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:55.650794 11835 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:55.675387 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.676247 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1220) took 1172 ms. Trace:
I20240507 08:05:55.676666 10261 rpcz_store.cc:276] 0507 08:05:54.503939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.667077 (+1163138us) service_pool.cc:225] Handling call
0507 08:05:55.676188 (+ 9111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:05:55.671031 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.661520 11538 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.650501 11744 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.679163 10264 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
W20240507 08:05:55.715945 11795 log.cc:924] Time spent T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 1.334s user 0.005s sys 0.000s
W20240507 08:05:55.716881 11874 log.cc:924] Time spent T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240507 08:05:55.674386 11833 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.740036 11833 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.681818 11878 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240507 08:05:55.741501 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39118 (request call id 53) took 1055 ms. Trace:
I20240507 08:05:55.741969 10263 rpcz_store.cc:276] 0507 08:05:54.685916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.686064 (+ 148us) service_pool.cc:225] Handling call
0507 08:05:55.741345 (+1055281us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b54002adad4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:55.741452 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1054335360}
I20240507 08:05:55.742911 11625 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:55.743078 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.751065 10262 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:05:55.753998 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1293) took 1107 ms. Trace:
I20240507 08:05:55.754400 10188 rpcz_store.cc:276] 0507 08:05:54.646535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.674744 (+ 28209us) service_pool.cc:225] Handling call
0507 08:05:54.675054 (+ 310us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:54.675398 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:54.675426 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:54.675849 (+ 423us) log.cc:842] Serialized 26 byte log entry
0507 08:05:54.724259 (+ 48410us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:05:54.724313 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:54.724403 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:55.752466 (+1028063us) spinlock_profiling.cc:243] Waited 987 us on lock 0x7b24001763d8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dd58024 00007fcf8dd5613b 00007fcf8dde4149 00007fcf8dde3dd5 00007fcf8dde3330 00007fcf90ad63a8 00007fcf90ad6b08 00007fcf90aae618 00007fcf90aa9a1a 00007fcf90aa8e66
0507 08:05:55.753680 (+ 1214us) raft_consensus.cc:1671] finished
0507 08:05:55.753704 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:55.753917 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1287,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":2270976,"thread_start_us":48038,"threads_started":1,"wal-append.queue_time_us":136837}
I20240507 08:05:55.756390 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:55.756927 10188 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:55.679769 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.720589 11787 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:55.721669 11773 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.722358 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:55.762444 10186 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 1.
I20240507 08:05:55.763072 11712 raft_consensus.cc:979] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:05:55.763181 11855 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.763603 11712 raft_consensus.cc:1067] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:05:55.763675 11855 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.764732 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 951) took 1235 ms. Trace:
I20240507 08:05:55.765077 10183 rpcz_store.cc:276] 0507 08:05:54.529629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.535596 (+ 5967us) service_pool.cc:225] Handling call
0507 08:05:54.535918 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:54.614306 (+ 78388us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:55.764523 (+1150217us) spinlock_profiling.cc:243] Waited 68.9 ms on lock 0x7b54002dc8d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:55.764685 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158527360}
I20240507 08:05:55.765821 11855 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.767678 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.768317 10259 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:55.769138 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.770153 11744 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.772653 10259 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:55.774313 10188 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:55.676168 11869 consensus_queue.cc:1035] T 7e6f917a0bed4f6289c1abc13f6260b9 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
W20240507 08:05:55.682057 11810 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.005s sys 0.039s
W20240507 08:05:55.775599 11827 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.821s user 0.002s sys 0.000s
I20240507 08:05:55.783257 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1294) took 1136 ms. Trace:
I20240507 08:05:55.783608 10187 rpcz_store.cc:276] 0507 08:05:54.646792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.685552 (+ 38760us) service_pool.cc:225] Handling call
0507 08:05:54.685947 (+ 395us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:54.694303 (+ 8356us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:54.694347 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:54.709973 (+ 15626us) log.cc:842] Serialized 26 byte log entry
0507 08:05:54.807540 (+ 97567us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:54.807606 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:54.807699 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:55.770906 (+963207us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:55.782951 (+ 12045us) raft_consensus.cc:1671] finished
0507 08:05:55.782986 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:55.783200 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5856,"raft.queue_time_us":3135,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":13436,"spinlock_wait_cycles":3889024,"thread_start_us":36315,"threads_started":1,"wal-append.queue_time_us":154907}
W20240507 08:05:55.789364 11810 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s user 0.005s sys 0.042s
I20240507 08:05:55.789383 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.799302 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.800801 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.801956 10183 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:55.801209 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 979) took 1117 ms. Trace:
I20240507 08:05:55.803965 10340 rpcz_store.cc:276] 0507 08:05:54.683348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.684822 (+ 1474us) service_pool.cc:225] Handling call
0507 08:05:54.732628 (+ 47806us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:55.499112 (+766484us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:55.800899 (+301787us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540014ebd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:55.801120 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568803200}
I20240507 08:05:55.807641 11736 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:55.808991 10259 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
W20240507 08:05:55.815814 10266 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1208) took 1768 ms (client timeout 1699 ms). Trace:
I20240507 08:05:55.816623 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:05:55.816391 10266 rpcz_store.cc:269] 0507 08:05:54.048303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.436910 (+388607us) service_pool.cc:225] Handling call
0507 08:05:55.815580 (+1378670us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b54001d2254. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:55.815712 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1005378304}
I20240507 08:05:55.817425 10259 raft_consensus.cc:2354] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:55.818859 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.820168 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.821768 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1232) took 1264 ms. Trace:
I20240507 08:05:55.823817 10258 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:55.731109 10260 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240507 08:05:55.770870 11538 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:55.838863 11816 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 1.023s user 0.002s sys 0.000s
I20240507 08:05:55.840970 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.843911 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.861881 10265 rpcz_store.cc:276] 0507 08:05:54.557385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.560554 (+ 3169us) service_pool.cc:225] Handling call
0507 08:05:55.821613 (+1261059us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b54001d1d54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:55.821708 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":341680896}
I20240507 08:05:55.884030 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.902678 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1242) took 1297 ms. Trace:
W20240507 08:05:55.902902 11783 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20240507 08:05:55.903580 10260 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:55.903133 10258 rpcz_store.cc:276] 0507 08:05:54.605422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.622650 (+ 17228us) service_pool.cc:225] Handling call
0507 08:05:55.823133 (+1200483us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:55.861994 (+ 38861us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:55.902432 (+ 40438us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54001d1d50. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:55.902595 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2710298496}
I20240507 08:05:55.904342 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.905102 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.905537 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.905714 10258 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:55.906339 10262 raft_consensus.cc:2379] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:55.906595 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.907084 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.907702 10262 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:55.907727 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.908480 10258 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:05:55.908653 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.901796 10267 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:05:55.910300 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.911224 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.902766 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.911923 11538 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:55.913149 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.913805 10266 raft_consensus.cc:2396] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20240507 08:05:55.912134 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.912597 10265 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
W20240507 08:05:55.916741 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 321955us
I20240507 08:05:55.916908 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.903409 11799 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:55.917856 11799 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.897778 10264 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:05:55.919788 11799 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:55.920924 10264 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:55.922653 11776 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:55.923750 10267 raft_consensus.cc:2354] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:55.924687 10267 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:55.924751 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:55.914697 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.915031 10260 raft_consensus.cc:2396] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:55.909297 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:55.925905 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:55.926510 10188 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:55.925370 10183 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:05:55.925927 10267 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:05:55.936321 10264 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:55.939343 10258 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:55.944301 10259 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:55.948592 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:55.957854 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 372249us
I20240507 08:05:55.959224 10264 raft_consensus.cc:1261] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240507 08:05:55.965832 10264 pending_rounds.cc:76] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 1
I20240507 08:05:55.966028 10264 pending_rounds.cc:97] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240507 08:05:55.966260 10264 raft_consensus.cc:2866] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:55.957615 10155 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:55.972770 10260 raft_consensus.cc:3032] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:55.973230 10260 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:55.974083 10260 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:55.975203 10260 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:55.977741 10262 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:55.977757 10335 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:55.981011 10264 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240507 08:05:55.997467 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 374410us
I20240507 08:05:55.952188 11848 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:56.060261 10306 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:56.061156 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:56.080888 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 123443us
I20240507 08:05:56.084168 10155 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:56.082365 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:55.955330 11846 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:55.977339 11750 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:56.082846 10186 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55820
W20240507 08:05:56.123508 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 63384.4us
I20240507 08:05:56.128584 10265 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:56.056007 11799 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:56.135835 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 53176.8us
I20240507 08:05:56.082129 11737 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:56.131968 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:56.127514 10186 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:05:56.147856 10186 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:56.056154 11777 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:55926 (request call id 2) took 2432 ms. Trace:
I20240507 08:05:56.144246 10259 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:56.094940 11601 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:05:56.129618 10232 leader_election.cc:334] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.551s (SENT)
I20240507 08:05:56.127537 11866 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:05:56.126082 11750 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:56.134279 11849 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:56.183601 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:56.184247 10183 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
W20240507 08:05:56.178980 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 83459.7us
I20240507 08:05:56.185056 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:56.186159 10155 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.138424 11813 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:56.140022 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:56.140419 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 87702.9us
W20240507 08:05:56.190416 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 60871.3us
I20240507 08:05:56.203145 10232 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:05:56.191068 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 51515.6us
I20240507 08:05:56.191015 10183 raft_consensus.cc:2379] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
W20240507 08:05:56.188803 10190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.005s sys 0.009s
I20240507 08:05:56.146739 11846 raft_consensus.cc:686] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:56.200675 10259 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42910
I20240507 08:05:56.206599 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:56.209286 10190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.005s sys 0.010s
I20240507 08:05:56.213878 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:56.180243 11750 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:56.217800 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 65710us
I20240507 08:05:56.221693 10183 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240507 08:05:56.153153 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
W20240507 08:05:56.224388 10337 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.056s sys 0.069s
W20240507 08:05:56.193565 10184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.018s sys 0.030s
I20240507 08:05:56.197944 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:56.153507 11866 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:56.142877 11830 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20240507 08:05:56.133905 11848 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.146574 11775 consensus_queue.cc:1046] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20240507 08:05:56.192090 11813 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:56.144249 11799 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:56.191244 10909 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.012s sys 0.002s
W20240507 08:05:56.220331 10254 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.634s user 0.026s sys 0.076s
W20240507 08:05:56.225222 10337 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.056s sys 0.069s
I20240507 08:05:56.227716 10183 raft_consensus.cc:1261] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:56.230172 10906 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.151719 11737 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.214507 10340 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 4
I20240507 08:05:56.218465 11811 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240507 08:05:56.261687 10304 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.033s sys 0.074s
W20240507 08:05:56.262118 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48886.1us
I20240507 08:05:56.263096 10261 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:56.220571 11852 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:56.222436 11579 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240507 08:05:56.207370 11754 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.399s user 0.006s sys 0.011s
W20240507 08:05:56.388402 11754 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.590s user 0.007s sys 0.014s
I20240507 08:05:56.266151 10188 raft_consensus.cc:1226] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
W20240507 08:05:56.389860 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 159826us
W20240507 08:05:56.225456 11738 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.960s user 0.022s sys 0.036s
W20240507 08:05:56.391171 11738 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.126s user 0.023s sys 0.036s
W20240507 08:05:56.231127 10184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s user 0.019s sys 0.030s
I20240507 08:05:56.269068 10186 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.393702 10186 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.395020 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:56.395655 10337 raft_consensus.cc:3032] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:56.395721 10186 raft_consensus.cc:1226] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:56.395982 10337 raft_consensus.cc:729] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:56.258877 11866 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:56.396675 10337 consensus_queue.cc:260] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.400801 10337 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:56.340524 10155 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.403381 11738 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:56.403456 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 143586us
I20240507 08:05:56.404507 10155 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:56.340376 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 81262.4us
I20240507 08:05:56.405681 10155 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:56.406675 10155 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:56.362426 10183 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:56.224231 11751 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20240507 08:05:56.408492 10340 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
W20240507 08:05:56.340379 11883 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20240507 08:05:56.348012 10909 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.012s sys 0.003s
I20240507 08:05:56.409039 11833 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:56.230479 10934 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 4
I20240507 08:05:56.410025 11878 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:56.410238 11814 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:56.410521 11878 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:56.412566 11878 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.407311 11754 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:56.409523 11833 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:56.414583 11833 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:56.415426 11881 log.cc:924] Time spent T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20240507 08:05:56.407948 11899 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:05:56.358685 10254 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.773s user 0.026s sys 0.076s
W20240507 08:05:56.374174 10304 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.781s user 0.033s sys 0.074s
W20240507 08:05:56.429925 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 842020us
W20240507 08:05:56.430564 10304 leader_election.cc:334] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.686s (SENT)
I20240507 08:05:56.418066 10156 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:56.339359 11813 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:56.265266 11877 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20240507 08:05:56.436146 10186 raft_consensus.cc:1261] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:56.425253 11899 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:56.438380 11899 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:56.438951 11741 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:56.439476 11741 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:56.270891 11796 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.157s
W20240507 08:05:56.475248 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 68705.4us
W20240507 08:05:56.479346 10304 leader_election.cc:334] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.806s (SENT)
I20240507 08:05:56.379309 11852 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:56.379657 11811 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:56.275796 10908 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
W20240507 08:05:56.502300 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 60206.5us
I20240507 08:05:56.359274 11744 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20240507 08:05:56.426545 11745 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:56.483821 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 77354.6us
I20240507 08:05:56.493476 10155 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:56.495878 10264 raft_consensus.cc:2932] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:05:56.521178 11882 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.426s user 0.000s sys 0.001s
I20240507 08:05:56.503055 11653 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:56.520720 11811 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 2
W20240507 08:05:56.565760 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 72505.1us
I20240507 08:05:56.521123 11852 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
W20240507 08:05:56.544922 11886 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.281s user 0.000s sys 0.001s
I20240507 08:05:56.366432 11846 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:56.440310 11741 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.503216 11736 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20240507 08:05:56.608274 10155 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:56.612701 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 176981us
W20240507 08:05:56.614598 10304 leader_election.cc:334] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.830s (SENT)
I20240507 08:05:56.615228 10304 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.565336 11830 raft_consensus.cc:2741] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240507 08:05:56.150007 11777 rpcz_store.cc:276] 0507 08:05:53.544743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:53.550584 (+ 5841us) service_pool.cc:225] Handling call
0507 08:05:53.639692 (+ 89108us) log.cc:842] Serialized 501 byte log entry
0507 08:05:54.146671 (+506979us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b18002eff00. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd1b8 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:05:55.977075 (+1830404us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8986,"raft.queue_time_us":29812,"raft.run_cpu_time_us":892,"raft.run_wall_time_us":67028,"spinlock_wait_cycles":13612800,"thread_start_us":329359,"threads_started":2,"wal-append.queue_time_us":259785,"wal-append.run_cpu_time_us":1754,"wal-append.run_wall_time_us":1091847}
W20240507 08:05:56.667294 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 98952.5us
W20240507 08:05:56.658756 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 62094.5us
I20240507 08:05:56.682276 11696 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:56.682545 11860 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:56.683001 11860 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.685070 11860 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.686877 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:56.687531 10339 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
W20240507 08:05:56.690127 11896 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
I20240507 08:05:56.691428 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1317) took 1219 ms. Trace:
W20240507 08:05:56.691712 11903 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20240507 08:05:56.691768 10190 rpcz_store.cc:276] 0507 08:05:55.471601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.471877 (+ 276us) service_pool.cc:225] Handling call
0507 08:05:55.491788 (+ 19911us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:55.527546 (+ 35758us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:55.527586 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:55.561201 (+ 33615us) log.cc:842] Serialized 26 byte log entry
0507 08:05:56.210622 (+649421us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:56.210674 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:56.210758 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:56.691187 (+480429us) raft_consensus.cc:1671] finished
0507 08:05:56.691212 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:56.691354 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":42958,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":10971,"thread_start_us":623087,"threads_started":2,"wal-append.queue_time_us":681712}
I20240507 08:05:56.693176 10337 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:56.694255 11736 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:56.695559 10934 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:56.576447 11892 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:56.699492 10155 leader_election.cc:398] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
W20240507 08:05:56.595861 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 329326us
I20240507 08:05:56.558445 11745 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:05:56.668109 11854 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240507 08:05:56.723202 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 108676us
I20240507 08:05:56.571978 11653 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.739032 10156 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:56.741690 10155 leader_election.cc:401] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20240507 08:05:56.753795 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 54512.4us
I20240507 08:05:56.754547 10155 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.755656 10155 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.713757 11744 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:56.752733 10304 leader_election.cc:334] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.849s (SENT)
I20240507 08:05:56.757123 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.767997 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.740625 11869 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:05:56.779589 10304 leader_election.cc:334] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.887s (SENT)
I20240507 08:05:56.780169 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.746181 11799 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.750077 11737 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:56.781548 11902 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20240507 08:05:56.777611 11830 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:56.744011 11736 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.778645 11753 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:56.763626 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:56.813542 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 45736.2us
I20240507 08:05:56.782315 11854 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:56.792737 11814 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:56.829303 11814 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:56.819309 10155 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.808763 11799 consensus_queue.cc:1046] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240507 08:05:56.823664 11880 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:56.830874 11880 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.833022 11880 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.824030 11890 tablet_replica.cc:330] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:05:56.785910 11866 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.781993 10137 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f7d05e73be847e99048092c9570faed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:05:56.835083 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1316) took 1363 ms. Trace:
W20240507 08:05:56.835248 10137 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 65099.6us
I20240507 08:05:56.836148 11783 raft_consensus.cc:3032] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:56.835458 10185 rpcz_store.cc:276] 0507 08:05:55.471235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.471987 (+ 752us) service_pool.cc:225] Handling call
0507 08:05:55.472332 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:55.975848 (+503516us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:55.975890 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:56.080841 (+104951us) log.cc:842] Serialized 26 byte log entry
0507 08:05:56.475412 (+394571us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:56.475460 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:56.475546 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:56.834872 (+359326us) raft_consensus.cc:1671] finished
0507 08:05:56.834896 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:56.835020 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20005,"raft.queue_time_us":55348,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":771761,"threads_started":2,"wal-append.queue_time_us":602994}
I20240507 08:05:56.836503 11783 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:56.837342 11783 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.839368 11866 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.841166 10187 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:56.824543 10340 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:56.824955 11737 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:05:56.825448 11908 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20240507 08:05:56.828603 11776 consensus_meta.cc:287] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.906s user 0.004s sys 0.005s
I20240507 08:05:56.808398 11775 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:56.771276 11744 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:56.849110 10304 leader_election.cc:334] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.983s (SENT)
I20240507 08:05:56.744776 11892 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.853157 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:56.821559 11900 log.cc:924] Time spent T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.292s user 0.000s sys 0.001s
I20240507 08:05:56.856352 10185 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:05:56.819343 11738 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:56.859472 10304 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.819517 11860 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:56.845932 11796 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:05:56.840021 11848 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.870652 10155 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:56.811292 10907 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:56.874305 10155 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:05:56.880546 10261 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.506s user 0.007s sys 0.000s
I20240507 08:05:56.880895 10261 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:56.881253 10259 raft_consensus.cc:595] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Received request to transfer leadership
I20240507 08:05:56.881520 10259 raft_consensus.cc:603] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:05:56.859139 11854 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:56.885994 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:56.887295 11877 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:05:56.887635 11653 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.888502 11892 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.877468 11796 raft_consensus.cc:686] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:56.889979 11796 consensus_queue.cc:237] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:56.891599 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 930) took 2743 ms (client timeout 1551 ms). Trace:
W20240507 08:05:56.892069 10191 rpcz_store.cc:269] 0507 08:05:54.148058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.148825 (+ 767us) service_pool.cc:225] Handling call
0507 08:05:56.891315 (+2742490us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b5400260554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:56.891502 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5048675584}
I20240507 08:05:56.892560 11811 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.850203 11737 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:56.893967 11904 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20240507 08:05:56.895033 11909 log.cc:924] Time spent T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20240507 08:05:56.895552 11737 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:56.886611 11538 consensus_meta.cc:287] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.969s user 0.008s sys 0.003s
I20240507 08:05:56.896756 10263 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:56.898823 11736 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.887682 11877 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:56.911089 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 99459.7us
I20240507 08:05:56.893909 10192 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:56.878448 11861 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:05:56.928563 11890 raft_consensus.cc:2229] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:05:56.911787 11848 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:56.938472 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:56.934415 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 383397us
I20240507 08:05:56.913422 11877 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:05:56.911533 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 85528.8us
I20240507 08:05:56.946228 11914 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:56.949000 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:56.952613 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:56.953075 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:56.933748 11754 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:05:56.925102 11825 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:56.997073 11783 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:56.931994 11813 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:56.966156 10263 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 919) took 1472 ms (client timeout 1833 ms). Trace:
I20240507 08:05:56.966554 10262 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240507 08:05:57.002630 10263 rpcz_store.cc:269] 0507 08:05:55.493667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.742822 (+249155us) service_pool.cc:225] Handling call
0507 08:05:56.965994 (+1223172us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400116554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:56.966103 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2652419456}
I20240507 08:05:56.966990 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 963) took 1381 ms. Trace:
I20240507 08:05:56.968251 11775 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:56.934093 11861 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:56.940698 11788 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:56.970938 11744 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.971652 11768 raft_consensus.cc:3032] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:05:56.972577 11890 raft_consensus.cc:2258] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:05:56.953256 10338 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
W20240507 08:05:56.969370 11899 consensus_meta.cc:287] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.531s user 0.002s sys 0.007s
W20240507 08:05:56.985961 11884 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.421s user 0.002s sys 0.000s
W20240507 08:05:56.987146 11833 consensus_meta.cc:287] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.572s user 0.005s sys 0.005s
I20240507 08:05:56.987795 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 945) took 2749 ms. Trace:
I20240507 08:05:56.987998 11811 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:56.988444 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 48417.5us
W20240507 08:05:56.989185 11750 consensus_meta.cc:287] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.761s user 0.006s sys 0.001s
I20240507 08:05:56.990178 11852 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:56.991060 11745 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:56.992065 11754 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:05:57.007023 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 84802.1us
I20240507 08:05:57.032805 10192 rpcz_store.cc:276] 0507 08:05:54.238699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.374605 (+135906us) service_pool.cc:225] Handling call
0507 08:05:56.893690 (+2519085us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:56.913848 (+ 20158us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:56.987520 (+ 73672us) spinlock_profiling.cc:243] Waited 2.5 s on lock 0x7b5400260550. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:56.987703 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5758195968}
I20240507 08:05:56.965559 11877 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.005105 11775 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:57.068197 11901 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.195s user 0.001s sys 0.001s
I20240507 08:05:57.068441 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:57.005136 11907 log.cc:924] Time spent T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.160s user 0.002s sys 0.000s
I20240507 08:05:57.055559 11754 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:05:57.038278 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 318464us
W20240507 08:05:57.038694 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 294382us
I20240507 08:05:57.074354 11825 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:57.063434 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.005491 11861 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:57.068207 11852 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.077859 11745 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.078372 11745 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.080725 11745 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:57.036096 11833 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.035111 11899 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.082469 10188 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:57.083233 11899 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.054492 10155 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:56.952163 10156 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:57.058148 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 667770us
I20240507 08:05:57.011626 11788 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.061234 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.094149 10262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1246) took 1495 ms. Trace:
I20240507 08:05:57.094951 10183 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:05:57.061328 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.096385 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 961) took 1558 ms. Trace:
I20240507 08:05:57.097761 11799 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.096756 10189 rpcz_store.cc:276] 0507 08:05:55.537943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.545910 (+ 7967us) service_pool.cc:225] Handling call
0507 08:05:55.546189 (+ 279us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:55.553729 (+ 7540us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:55.553787 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:55.554308 (+ 521us) log.cc:842] Serialized 26 byte log entry
0507 08:05:55.923393 (+369085us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:55.923447 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:55.923565 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:56.985277 (+1061712us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:57.084781 (+ 99504us) raft_consensus.cc:1671] finished
0507 08:05:57.084804 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:57.096328 (+ 11524us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7189,"raft.queue_time_us":8644,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":97,"spinlock_wait_cycles":8077696,"thread_start_us":368689,"threads_started":1,"wal-append.queue_time_us":966528}
I20240507 08:05:57.061604 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.098875 10187 raft_consensus.cc:2396] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20240507 08:05:57.065829 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.011538 11768 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:57.094224 11744 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.082243 11657 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:57.094660 10262 rpcz_store.cc:276] 0507 08:05:55.598922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:56.082365 (+483443us) service_pool.cc:225] Handling call
0507 08:05:56.966126 (+883761us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:57.002770 (+ 36644us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:57.093972 (+ 91202us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b5400116550. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:57.094097 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1921435904}
W20240507 08:05:57.151669 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 97340.9us
W20240507 08:05:57.099854 10261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 956) took 1546 ms (client timeout 1723 ms). Trace:
W20240507 08:05:57.191061 10261 rpcz_store.cc:269] 0507 08:05:55.553159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.678862 (+125703us) service_pool.cc:225] Handling call
0507 08:05:57.099679 (+1420817us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b540035a054. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:57.099782 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1158854912}
I20240507 08:05:57.158215 10158 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:57.161592 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 209568us
I20240507 08:05:57.004307 10184 rpcz_store.cc:276] 0507 08:05:55.571741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.582493 (+ 10752us) service_pool.cc:225] Handling call
0507 08:05:55.616420 (+ 33927us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:55.629349 (+ 12929us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:55.630891 (+ 1542us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:55.653305 (+ 22414us) log.cc:842] Serialized 26 byte log entry
0507 08:05:56.391863 (+738558us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:56.391912 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:56.392019 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:56.909257 (+517238us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:56.933083 (+ 23826us) raft_consensus.cc:1671] finished
0507 08:05:56.933124 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:56.952828 (+ 19704us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46062336,"thread_start_us":738182,"threads_started":1,"wal-append.queue_time_us":1216637}
I20240507 08:05:57.182116 10192 raft_consensus.cc:2396] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20240507 08:05:57.161708 11837 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.970s user 0.005s sys 0.014s
I20240507 08:05:57.217960 10156 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:57.039252 11750 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.083777 11696 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.114s
I20240507 08:05:57.232571 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.162499 11924 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:57.191134 11654 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.236987 11696 consensus_queue.cc:1035] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:57.237864 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.236919 11879 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240507 08:05:57.240999 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:57.266955 11750 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.241767 11775 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.151741 11799 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.225271 11880 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:57.245996 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.190347 11744 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.247049 11833 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:57.248211 11906 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.359s user 0.000s sys 0.001s
I20240507 08:05:57.195263 11745 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.198400 11848 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:05:57.249672 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 161479us
W20240507 08:05:57.224268 11837 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.040s user 0.005s sys 0.014s
I20240507 08:05:57.232868 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:57.318078 10183 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:57.201297 11852 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.263911 11811 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.284947 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:42910 (request call id 456) took 1727 ms. Trace:
I20240507 08:05:57.289602 11860 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:57.291275 11879 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:57.270285 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:57.264984 11654 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:57.233355 11830 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.217675 11813 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:57.316246 10304 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.319262 10942 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:57.319515 10259 rpcz_store.cc:276] 0507 08:05:55.558172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:56.200383 (+642211us) service_pool.cc:225] Handling call
0507 08:05:57.284721 (+1084338us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b540035a054. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e21e13 00007fcf95d6c5ba 00007fcf8f45bd14 00007fcf8f45bca2 00007fcf8f45bbd1 00007fcf8f45bb4c 00007fcf8f45ae02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:57.284856 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1547255552}
I20240507 08:05:57.321472 10184 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20240507 08:05:57.249095 11712 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20240507 08:05:57.307832 11889 log.cc:924] Time spent T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240507 08:05:57.296873 11768 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.354321 11768 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:57.354737 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.323920 11879 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:57.325253 10186 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:57.353619 11775 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:57.353806 11890 ts_tablet_manager.cc:1875] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240507 08:05:57.325953 11887 log.cc:924] Time spent T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.289s user 0.001s sys 0.001s
W20240507 08:05:57.364589 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 48442.3us
I20240507 08:05:57.322183 11848 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:05:57.367548 10258 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.019s sys 0.007s
I20240507 08:05:57.369082 10304 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.376818 10233 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
W20240507 08:05:57.380874 10258 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.818s user 0.019s sys 0.007s
I20240507 08:05:57.350340 11654 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.351337 11830 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:57.323602 11811 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.316941 11745 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.431059 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:57.435792 10184 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:57.365715 11813 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:05:57.315987 11880 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.446870 10335 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:57.323163 11852 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.455457 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 76) took 1453 ms. Trace:
I20240507 08:05:57.458809 10258 rpcz_store.cc:276] 0507 08:05:56.001893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:56.129055 (+127162us) service_pool.cc:225] Handling call
0507 08:05:56.129396 (+ 341us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:56.139926 (+ 10530us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:56.139956 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:56.140331 (+ 375us) log.cc:842] Serialized 26 byte log entry
0507 08:05:56.537406 (+397075us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:05:56.993167 (+455761us) spinlock_profiling.cc:243] Waited 62.6 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:57.402650 (+409483us) log.cc:842] Serialized 14 byte log entry
0507 08:05:57.403005 (+ 355us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:57.403094 (+ 89us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:57.403153 (+ 59us) raft_consensus.cc:1671] finished
0507 08:05:57.455209 (+ 52056us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:57.455388 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":416898,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":4284,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":45966,"spinlock_wait_cycles":144020864,"thread_start_us":1155813,"threads_started":2,"wal-append.queue_time_us":626517}
W20240507 08:05:57.462859 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 93914.5us
I20240507 08:05:57.465699 10183 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:05:57.469887 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:57.470611 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 87) took 1329 ms. Trace:
I20240507 08:05:57.471065 10335 rpcz_store.cc:276] 0507 08:05:56.140975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:56.141108 (+ 133us) service_pool.cc:225] Handling call
0507 08:05:56.151488 (+ 10380us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:57.447330 (+1295842us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:57.470344 (+ 23014us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540013e554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:57.470541 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2979892608}
I20240507 08:05:57.409698 11737 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.481007 10304 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.352510 11754 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:57.456143 11906 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20240507 08:05:57.431615 11830 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.430400 11923 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.479509 11914 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.456926 11813 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.358479 11879 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.476400 11736 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:57.463624 11918 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:57.502204 10138 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
I20240507 08:05:57.454571 11877 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.462816 11924 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:57.502664 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:57.482663 11824 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.004s sys 0.022s
I20240507 08:05:57.503234 11799 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.504065 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 79) took 1501 ms. Trace:
I20240507 08:05:57.514688 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.524232 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:57.541904 10335 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:57.526109 10184 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
W20240507 08:05:57.524456 10337 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.007s sys 0.014s
W20240507 08:05:57.527995 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47083.4us
W20240507 08:05:57.536600 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20240507 08:05:57.430845 11861 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.543370 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.539847 10264 rpcz_store.cc:276] 0507 08:05:56.002699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:56.135274 (+132575us) service_pool.cc:225] Handling call
0507 08:05:56.135860 (+ 586us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:56.136232 (+ 372us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:56.136269 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:56.148952 (+ 12683us) log.cc:842] Serialized 501 byte log entry
0507 08:05:56.495749 (+346797us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:05:56.599664 (+103915us) log.cc:842] Serialized 14 byte log entry
0507 08:05:56.888929 (+289265us) spinlock_profiling.cc:243] Waited 4.47 ms on lock 0x7b180030bb00. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd1b8 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:05:57.217315 (+328386us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:57.292230 (+ 74915us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b180030bb00. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf90ab675c 00007fcf8dedf54c 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8def9cc5 00007fcf90e1f620 00007fcf90e317c6 00007fcf90e42327
0507 08:05:57.292356 (+ 126us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:57.292446 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:57.503835 (+211389us) raft_consensus.cc:1671] finished
0507 08:05:57.503857 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:57.503994 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":209798,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":7101,"spinlock_wait_cycles":75448320,"thread_start_us":537224,"threads_started":2,"wal-append.queue_time_us":655340}
I20240507 08:05:57.506453 11923 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.511976 11813 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:57.550115 10304 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.455912 11915 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.107s
W20240507 08:05:57.544505 10337 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.533s user 0.007s sys 0.014s
I20240507 08:05:57.509140 11788 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:57.550760 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 324655us
I20240507 08:05:57.559038 10155 leader_election.cc:398] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:57.559422 10155 leader_election.cc:401] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20240507 08:05:57.527609 11712 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:57.523594 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:05:57.545267 11824 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s user 0.005s sys 0.022s
I20240507 08:05:57.506121 11737 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.576458 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.585361 10183 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:57.511440 11914 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.593477 10232 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:57.553262 11696 consensus_queue.cc:1046] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20240507 08:05:57.557062 11923 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:57.602937 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 58749.6us
I20240507 08:05:57.614890 10304 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.572557 11712 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.582212 10340 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:57.601081 11796 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.032s sys 0.024s
I20240507 08:05:57.580943 11925 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3.
I20240507 08:05:57.629961 10335 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:05:57.570734 11736 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:57.579921 11949 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.544476 11918 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.639820 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:05:57.631428 11796 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s user 0.032s sys 0.025s
I20240507 08:05:57.644754 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.545804 11924 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.619634 11744 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.627528 11712 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:05:57.666218 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 51435us
I20240507 08:05:57.626821 11778 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.527837 11848 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.632858 11538 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.638856 11783 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:57.677466 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.640048 11824 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:57.675516 10304 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.634193 11750 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
W20240507 08:05:57.687165 11944 log.cc:924] Time spent T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20240507 08:05:57.688997 11918 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.674228 11745 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.674386 11538 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:57.690093 11754 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.679087 11623 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.680718 11811 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.691159 11914 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.690910 11623 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.693143 11623 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.697459 11924 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.701752 10942 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20240507 08:05:57.702163 10942 pending_rounds.cc:76] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Aborting all ops after (but not including) 4
I20240507 08:05:57.702371 10942 pending_rounds.cc:97] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240507 08:05:57.702559 11750 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:57.703039 11750 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:57.703999 11750 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.706121 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.706976 11861 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.648414 11949 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.707291 10304 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.711194 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.711400 11937 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.711880 11937 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.712214 11937 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.712620 11799 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.714392 11915 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:57.714903 11915 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.716840 11915 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:57.707449 11824 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:57.716849 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.642954 11854 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20240507 08:05:57.720306 10304 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.720585 11837 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:57.720992 11778 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.721410 11778 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.723263 11825 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:57.724813 11825 consensus_queue.cc:1046] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:57.685060 10907 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:05:57.726505 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 47495.7us
I20240507 08:05:57.727491 11937 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.728799 11713 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:05:57.729280 11713 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 6 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:57.729705 11825 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240507 08:05:57.730137 11927 log.cc:924] Time spent T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20240507 08:05:57.730372 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.730590 11846 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:05:57.732364 11846 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:57.732718 11846 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.733085 11846 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.733754 11788 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:57.735066 10304 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.735003 11788 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:57.735888 11824 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:57.736796 11696 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:57.737038 11920 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:57.738732 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.736482 11824 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20240507 08:05:57.730327 11825 raft_consensus.cc:3032] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:57.740718 11825 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:57.709503 11949 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.729429 11937 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:57.741897 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.742172 10908 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.742846 10341 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:57.743525 11825 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.744846 11825 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:05:57.746292 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 64306.4us
I20240507 08:05:57.747274 10339 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:57.705796 11736 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.749322 10339 raft_consensus.cc:1226] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:57.702621 10942 raft_consensus.cc:2866] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:57.717459 11915 raft_consensus.cc:3032] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:57.766520 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.767644 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.768661 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.769867 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.705361 11750 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:57.781708 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.781949 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.783931 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:57.713133 11799 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.690121 11538 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.784890 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.787935 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.696497 11923 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.806245 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.806650 10333 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:57.807603 11923 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:57.771672 11915 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:57.808992 10267 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240507 08:05:57.764384 11768 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:57.809324 11915 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.809336 11949 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:57.810642 11915 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.810720 10260 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:57.797363 11948 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:57.730150 11713 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.801955 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:57.813997 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.814946 10264 raft_consensus.cc:2354] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:05:57.803114 11799 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.784261 11861 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20240507 08:05:57.815529 11957 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.816031 11957 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.811872 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.817477 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.811020 11949 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.817981 11957 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.818560 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.819145 10335 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:57.819355 11949 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:57.819936 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.820428 10335 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:57.821053 11846 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.822479 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.823180 10260 raft_consensus.cc:2379] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:57.809358 10267 pending_rounds.cc:76] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 2
I20240507 08:05:57.823889 10267 pending_rounds.cc:97] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240507 08:05:57.824172 10267 raft_consensus.cc:2866] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:57.824419 11829 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:57.824780 11829 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:57.825516 11829 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.823204 11846 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:57.826491 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:57.826711 11829 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.812034 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.714313 11880 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:57.812264 10155 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.802695 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.815304 11837 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:57.829705 10341 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:57.779551 11852 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.840181 10259 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:57.841637 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.839406 10336 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:57.844837 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.851787 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.852473 10262 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.853670 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.854239 10259 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.801034 11778 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.861518 10335 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:57.866156 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.869598 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.778419 11654 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.873899 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.875110 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.876248 11940 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:57.876777 11940 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.878286 10266 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240507 08:05:57.878640 10266 pending_rounds.cc:76] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 2
I20240507 08:05:57.878868 10266 pending_rounds.cc:97] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240507 08:05:57.879127 10266 raft_consensus.cc:2866] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:57.877630 10259 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:57.880200 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.880756 10266 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:57.880438 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.881827 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.882323 10259 raft_consensus.cc:2379] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:57.882455 10265 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:57.876168 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.883263 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.884006 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.881632 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.887794 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.888423 10259 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.890065 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.891721 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.892441 10338 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:57.834867 11949 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:57.681741 11925 consensus_queue.cc:1046] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.148s
I20240507 08:05:57.895236 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.896342 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.899600 10335 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:05:57.896751 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.900810 10260 raft_consensus.cc:2379] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:57.786736 11924 consensus_queue.cc:1035] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.065s
I20240507 08:05:57.868618 11853 raft_consensus.cc:2741] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240507 08:05:57.787171 11830 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.813923 11756 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.075s
I20240507 08:05:57.903076 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.905871 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.900285 10339 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:57.882619 10340 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:57.931373 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.931718 10259 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:57.937435 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.937567 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.938073 10259 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:57.938190 10335 raft_consensus.cc:2379] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:57.933768 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.939488 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.940428 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:57.943044 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 401873us
I20240507 08:05:57.944701 10259 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:05:57.934513 10265 raft_consensus.cc:1799] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Handling vote request from an unknown peer 59382e6647c549d89d83c70233c0667e
I20240507 08:05:57.854182 11813 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.948207 10187 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:57.946197 10265 raft_consensus.cc:2396] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240507 08:05:57.949973 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.952438 10338 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:05:57.946161 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1025) took 1004 ms. Trace:
I20240507 08:05:57.900317 11941 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:57.959333 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.969642 10259 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 2: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:57.970369 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.970929 10259 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:57.960114 10231 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.968524 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.974736 10304 leader_election.cc:398] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:05:57.974931 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.976315 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.978719 10231 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.979894 10231 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.980952 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.983492 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:57.984192 10263 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:05:57.984351 10231 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.986590 10332 raft_consensus.cc:1226] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:57.988971 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:57.989871 10335 raft_consensus.cc:2362] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3. Re-sending same reply.
I20240507 08:05:57.968158 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:57.991838 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:57.992559 10265 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:57.995138 10231 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.834225 11948 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:57.997893 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.900507 11949 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:57.958830 11814 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.014844 10231 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:57.974553 11937 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:58.021523 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.022450 10335 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 2: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:57.993541 11914 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.033859 10155 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:58.030265 10262 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:58.037500 10259 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:58.004333 11745 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.038978 10155 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:57.997761 11811 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.040015 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.043496 10262 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:57.972508 11923 consensus_queue.cc:1046] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20240507 08:05:58.051451 11962 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:58.041623 11937 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.045686 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.046355 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:58.174153 11654 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:57.984670 11824 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20240507 08:05:58.016904 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:57.997182 11696 consensus_queue.cc:1046] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20240507 08:05:58.018693 11657 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.112435 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.114346 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.120631 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.120793 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.121835 10186 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:05:58.122169 10183 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:58.122743 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.123014 11830 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.123214 11783 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:58.212427 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.123876 11813 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:58.124281 11925 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:58.212890 11783 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.213469 11925 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.213896 10185 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20240507 08:05:58.214331 11968 log.cc:924] Time spent T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20240507 08:05:58.214994 10185 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
W20240507 08:05:58.215242 11964 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240507 08:05:58.215140 11783 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:05:58.215684 11966 log.cc:924] Time spent T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20240507 08:05:58.125741 11778 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:58.129099 11967 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.264s user 0.002s sys 0.000s
I20240507 08:05:58.131156 11737 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.042243 11744 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.140923 10262 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:58.013463 11868 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:58.047295 11957 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:58.153702 10338 raft_consensus.cc:2396] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:05:58.165169 10158 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:58.165419 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 184618us
I20240507 08:05:57.981675 11941 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:58.225291 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 60244.5us
I20240507 08:05:58.117897 11738 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:58.050977 11754 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.119s
I20240507 08:05:58.165022 11811 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:58.007793 11918 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:58.241880 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.242337 11868 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.243141 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.244066 10190 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:58.244370 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.244875 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:05:58.200724 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 154487us
I20240507 08:05:58.201817 11846 raft_consensus.cc:3032] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:05:58.186008 11890 ts_tablet_manager.cc:1888] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240507 08:05:58.246491 10232 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.212747 11949 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.215579 11925 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.125744 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.186715 11940 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:58.237857 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.230376 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.198985 11962 raft_consensus.cc:686] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.239749 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:58.245983 10155 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:58.212261 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.202299 11848 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.251878 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.252198 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.252763 10232 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.254338 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.204010 11579 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:58.269764 10192 raft_consensus.cc:2379] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:58.269879 10155 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.271004 10155 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.272292 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.276309 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:58.259143 11890 log.cc:1196] T e9adf1535c974b2abf910f3c1c537627 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/e9adf1535c974b2abf910f3c1c537627
I20240507 08:05:58.253569 11811 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.238334 11830 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.210825 11538 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.258307 11745 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.266688 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:58.267226 11960 raft_consensus.cc:3032] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
W20240507 08:05:58.262640 11712 consensus_meta.cc:287] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.586s user 0.008s sys 0.001s
I20240507 08:05:58.278905 10304 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:58.278931 10192 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240507 08:05:58.298355 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 283716us
I20240507 08:05:58.299290 10231 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.278198 11925 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:58.250684 11846 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:58.258654 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.281869 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.282558 10264 pending_rounds.cc:76] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 1
I20240507 08:05:58.303992 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.271878 11848 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.320222 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.320933 10339 raft_consensus.cc:2362] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5. Re-sending same reply.
I20240507 08:05:58.277885 11579 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.289856 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.265321 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.290700 10188 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:05:58.253685 10266 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:05:58.290908 11741 consensus_queue.cc:1046] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:58.294204 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.295313 10262 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.245095 11868 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:58.281651 11754 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:05:58.279599 11654 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:58.304385 11712 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.289094 11756 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240507 08:05:58.344457 11756 raft_consensus.cc:3032] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:58.295123 11960 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:58.280100 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.281785 11957 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.350121 10341 raft_consensus.cc:2379] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:58.350217 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.351684 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.352344 10258 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.305030 10231 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.287950 11937 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:58.306363 10264 pending_rounds.cc:97] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 3.2
W20240507 08:05:58.266203 11969 log.cc:924] Time spent T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20240507 08:05:58.271803 11657 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20240507 08:05:58.341703 10306 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:58.305140 11829 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:58.376348 11829 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.378096 11829 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:05:58.353722 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48788.7us
I20240507 08:05:58.378911 10231 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.378989 11579 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.379573 11914 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.304014 11696 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:58.380126 11696 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:58.379956 11958 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.380519 11696 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:58.379961 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.301906 10934 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:58.355644 10264 raft_consensus.cc:2866] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:05:58.386482 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.387125 11985 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.344834 11756 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:58.390599 11756 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.391721 11756 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:58.356849 10156 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.357967 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.325323 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.323360 11745 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240507 08:05:58.373489 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 48454.9us
I20240507 08:05:58.349469 10333 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.375448 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.375834 11937 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:58.385391 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:58.356781 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.388135 10231 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.346685 11788 consensus_queue.cc:1046] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20240507 08:05:58.325323 11925 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.345039 11712 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:05:58.388773 11974 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240507 08:05:58.427943 10192 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:05:58.429291 10231 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.325722 11824 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20240507 08:05:58.358656 11957 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.354226 10262 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:58.357379 11890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42910 (request call id 455) took 2796 ms. Trace:
W20240507 08:05:58.412853 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56108.3us
I20240507 08:05:58.425135 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.432350 10231 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.433205 11811 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.433751 11736 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.436110 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.443823 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.438462 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.444337 10231 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.404074 11754 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.446296 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.452817 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.453028 11738 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.423810 10266 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:58.427225 11741 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.459005 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.431133 11960 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.444695 10192 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:58.469636 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.469841 10232 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:58.412453 11657 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20240507 08:05:58.455789 10231 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:58.458241 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.472635 10261 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:05:58.453439 10334 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 2.
I20240507 08:05:58.449446 11941 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.443715 10340 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:58.465466 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.479352 10188 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.445122 11745 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:58.466061 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.466873 10339 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.443464 11949 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:58.394750 11914 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.447656 11744 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.474040 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.359043 11962 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:58.479800 11977 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20240507 08:05:58.474736 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.491942 11654 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.492097 10334 raft_consensus.cc:1226] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:58.452847 11829 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240507 08:05:58.493999 10907 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:58.474112 11976 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:58.495229 11976 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:58.495538 11751 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:58.453864 11975 consensus_queue.cc:1046] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:58.393224 11846 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.465546 11811 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.458710 11830 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:58.459228 10189 raft_consensus.cc:1226] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:58.480173 10267 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:05:58.507046 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:05:58.483832 11981 log.cc:924] Time spent T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20240507 08:05:58.490082 10261 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:05:58.495607 11976 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:58.496294 10190 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:58.459527 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.514271 10335 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.475139 10231 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:58.480015 11750 consensus_meta.cc:287] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.693s user 0.005s sys 0.005s
I20240507 08:05:58.505160 10258 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:58.478240 11915 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:58.478628 11538 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.505853 10942 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240507 08:05:58.480556 10186 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:05:58.507666 10267 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:58.531996 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.508611 10339 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:58.533213 10339 raft_consensus.cc:1226] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.483006 11960 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:58.510385 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.496105 11751 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:58.538975 11751 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:58.539078 11657 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:58.539705 11657 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.542183 10156 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:58.517236 10231 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.476886 11696 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.506829 11788 raft_consensus.cc:3032] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:05:58.503320 11914 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:58.521781 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.552430 10258 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:58.553416 10258 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20240507 08:05:58.554569 10258 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240507 08:05:58.522336 10188 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:58.508262 11830 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.532411 10267 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.535275 10339 raft_consensus.cc:1261] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:58.497263 11949 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.520905 11811 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.555343 10261 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:58.541767 11657 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:58.445654 11890 rpcz_store.cc:276] 0507 08:05:55.556001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:55.556380 (+ 379us) service_pool.cc:225] Handling call
0507 08:05:56.577049 (+1020669us) ts_tablet_manager.cc:1120] Deleting tablet e9adf1535c974b2abf910f3c1c537627
0507 08:05:56.596209 (+ 19160us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:05:57.921440 (+1325231us) tablet_metadata.cc:641] Metadata flushed
0507 08:05:58.168761 (+247321us) tablet_metadata.cc:641] Metadata flushed
0507 08:05:58.345925 (+177164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":96872,"tablet-delete.queue_time_us":976344,"thread_start_us":840233,"threads_started":1}
I20240507 08:05:58.557194 11914 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.561125 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.562726 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.555953 10232 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:58.561450 11744 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20240507 08:05:58.568001 10339 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:58.551880 11860 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:58.568224 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1029) took 1073 ms. Trace:
I20240507 08:05:58.595393 10332 rpcz_store.cc:276] 0507 08:05:57.495206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:57.495345 (+ 139us) service_pool.cc:225] Handling call
0507 08:05:57.502395 (+ 7050us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:57.987081 (+484686us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.567952 (+580871us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b540013e054. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:58.568154 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1113435520}
I20240507 08:05:58.482712 11925 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.592609 10261 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:58.571622 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:58.542629 11915 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:58.601173 10942 raft_consensus.cc:2379] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:05:58.552102 11788 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:58.570559 10267 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:58.589864 10231 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.607120 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.605867 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.609720 10339 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:58.549546 11952 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20240507 08:05:58.553089 11941 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20240507 08:05:58.562211 11538 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.624318 10231 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.570672 11830 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:58.572655 11949 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.513005 11754 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.605406 11860 raft_consensus.cc:686] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.623349 10190 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:58.572547 11601 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:58.630280 10261 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:58.634599 10338 raft_consensus.cc:1226] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.622207 10340 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.605513 11745 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:58.601434 11811 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:58.637986 11985 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.627533 10339 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.641556 10231 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.626680 11736 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.641480 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 990) took 1390 ms. Trace:
I20240507 08:05:58.643703 10232 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:58.648005 10190 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:58.651450 10187 rpcz_store.cc:276] 0507 08:05:57.250670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:57.250854 (+ 184us) service_pool.cc:225] Handling call
0507 08:05:57.265958 (+ 15104us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:57.957904 (+691946us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.641284 (+683380us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540005ee54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:58.641426 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1569031936}
I20240507 08:05:58.651888 10231 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.653370 10231 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.653445 10339 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240507 08:05:58.605600 11849 consensus_queue.cc:1046] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20240507 08:05:58.654803 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.656805 11918 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:58.657327 11830 consensus_queue.cc:1046] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:58.657718 11737 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.640591 11788 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.658351 11737 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:58.658485 11830 raft_consensus.cc:3032] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:58.658914 11830 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.654003 11957 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.657281 11918 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:58.663780 11849 raft_consensus.cc:3032] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:58.663278 11830 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.664256 11849 raft_consensus.cc:729] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.661651 11899 consensus_queue.cc:1046] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20240507 08:05:58.666031 10231 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.666522 11899 raft_consensus.cc:3032] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:58.666884 11899 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.668201 11899 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.669272 11899 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:58.664746 11830 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:58.676681 10339 raft_consensus.cc:1261] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:58.679245 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.643985 11601 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.685148 10231 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.658706 11788 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:05:58.686883 11991 log.cc:924] Time spent T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.126s user 0.000s sys 0.001s
I20240507 08:05:58.689741 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.691077 10338 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:05:58.675905 11918 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:58.680945 11949 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.670420 11696 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:58.669874 11811 raft_consensus.cc:686] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:58.711607 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.665110 11849 consensus_queue.cc:260] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.655824 11745 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.671687 11936 consensus_queue.cc:1046] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240507 08:05:58.691919 11852 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.658915 11737 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.046s
I20240507 08:05:58.693228 10266 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:58.712731 11985 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.711500 11914 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.704638 11744 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:58.692652 11833 consensus_queue.cc:1046] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20240507 08:05:58.694880 11861 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.064s
I20240507 08:05:58.767616 11744 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.707569 11941 consensus_queue.cc:1035] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20240507 08:05:58.770159 11744 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:58.750166 11538 consensus_queue.cc:1046] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20240507 08:05:58.771518 11993 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20240507 08:05:58.772846 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:58.766459 10335 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.751143 11738 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:58.756678 11959 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:05:58.794803 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:58.692509 11915 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.756939 11976 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.810969 11959 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:58.802982 11985 consensus_queue.cc:1035] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
W20240507 08:05:58.834985 11996 log.cc:924] Time spent T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240507 08:05:58.787820 10333 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:58.842345 10261 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:05:58.850607 10334 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.778287 11860 consensus_queue.cc:237] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.793246 11738 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.823135 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:58.860409 10089 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 68) took 4674 ms. Trace:
I20240507 08:05:58.814280 11751 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.867122 11751 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.865180 10266 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:58.869946 11848 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:58.870184 10940 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:58.870342 11848 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.874969 11978 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:05:58.876075 11978 raft_consensus.cc:3032] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:58.876449 11978 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:05:58.876749 11999 log.cc:924] Time spent T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20240507 08:05:58.798173 11962 consensus_queue.cc:1046] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20240507 08:05:58.885154 10156 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:58.883181 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:05:58.799038 11768 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20240507 08:05:58.892722 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:58.850653 11745 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:58.810014 11811 consensus_queue.cc:237] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.902388 10259 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240507 08:05:58.908710 10259 raft_consensus.cc:1226] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:58.865135 10089 rpcz_store.cc:276] 0507 08:05:54.186340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:54.192595 (+ 6255us) service_pool.cc:225] Handling call
0507 08:05:58.860345 (+4667750us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:05:54.640119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:05:54.640783 (+ 664us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:05:54.640824 (+ 41us) write_op.cc:435] Acquired schema lock
0507 08:05:54.640838 (+ 14us) tablet.cc:660] Decoding operations
0507 08:05:54.641493 (+ 655us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:05:54.641567 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0507 08:05:54.641638 (+ 71us) tablet.cc:683] Acquiring locks for 3 operations
0507 08:05:54.642058 (+ 420us) tablet.cc:699] Row locks acquired
0507 08:05:54.642080 (+ 22us) write_op.cc:260] PREPARE: finished
0507 08:05:55.676292 (+1034212us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b2c000571b8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8f809f11 00007fcf8f818bda 00007fcf8f818863 00007fcf8f818801 00007fcf90e62ec2 00007fcf90e62c9d 00007fcf94838160 00007fcf94837c78 00007fcf9483c561 00007fcf9483c519 00007fcf9483c4a9
0507 08:05:55.676356 (+ 64us) write_op.cc:270] Start()
0507 08:05:55.678709 (+ 2353us) write_op.cc:276] Timestamp: P: 1715069154646449 usec, L: 0
0507 08:05:55.678740 (+ 31us) op_driver.cc:331] REPLICATION: starting
0507 08:05:55.785860 (+107120us) log.cc:842] Serialized 1301 byte log entry
0507 08:05:57.014980 (+1229120us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:57.732620 (+717640us) op_driver.cc:444] REPLICATION: finished
0507 08:05:58.151606 (+418986us) write_op.cc:301] APPLY: starting
0507 08:05:58.151692 (+ 86us) tablet.cc:1365] starting BulkCheckPresence
0507 08:05:58.151827 (+ 135us) tablet.cc:1368] finished BulkCheckPresence
0507 08:05:58.151854 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:05:58.216370 (+ 64516us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:05:58.216543 (+ 173us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0507 08:05:58.216588 (+ 45us) write_op.cc:312] APPLY: finished
0507 08:05:58.216801 (+ 213us) log.cc:842] Serialized 34 byte log entry
0507 08:05:58.216949 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:58.217045 (+ 96us) write_op.cc:454] Released schema lock
0507 08:05:58.217097 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0507 08:05:58.217362 (+ 265us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:05:58.217391 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":385223,"apply.run_cpu_time_us":2666,"apply.run_wall_time_us":66058,"num_ops":3,"prepare.queue_time_us":156482,"prepare.run_cpu_time_us":27973,"prepare.run_wall_time_us":1504174,"raft.queue_time_us":663289,"raft.run_cpu_time_us":53619,"raft.run_wall_time_us":218772,"replication_time_us":2053783,"spinlock_wait_cycles":87249536,"thread_start_us":1156700,"threads_started":4,"wal-append.queue_time_us":1059113}]]}
I20240507 08:05:58.897435 11833 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:05:58.914245 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 119601us
I20240507 08:05:58.877332 11978 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:58.922500 10259 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:58.925009 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:58.925253 11833 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:58.978600 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:58.982439 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1264) took 1025 ms. Trace:
I20240507 08:05:58.988802 11976 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:58.990671 10266 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:58.991765 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:58.991647 11833 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:58.992425 10266 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
W20240507 08:05:59.002502 12005 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240507 08:05:58.983076 10260 rpcz_store.cc:276] 0507 08:05:57.957010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.565908 (+608898us) service_pool.cc:225] Handling call
0507 08:05:58.566228 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.566566 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.566595 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.567039 (+ 444us) log.cc:842] Serialized 26 byte log entry
0507 08:05:58.645611 (+ 78572us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:58.660681 (+ 15070us) log.cc:842] Serialized 14 byte log entry
0507 08:05:58.675923 (+ 15242us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.676004 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:58.982144 (+306140us) raft_consensus.cc:1671] finished
0507 08:05:58.982171 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:58.982355 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22481,"prepare.queue_time_us":14935,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":16106,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":4183,"thread_start_us":78252,"threads_started":1,"wal-append.queue_time_us":120789}
I20240507 08:05:58.973798 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:59.012496 10333 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:05:59.013942 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:59.015565 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:59.015867 11738 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:59.016323 10333 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:59.020629 10266 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:59.024948 10187 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:05:59.027601 10260 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:59.027726 10231 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.034441 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:59.028826 12007 log.cc:924] Time spent T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240507 08:05:59.048051 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:59.049214 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:59.057744 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 961) took 1253 ms. Trace:
I20240507 08:05:59.058141 10341 rpcz_store.cc:276] 0507 08:05:57.804178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:57.875943 (+ 71765us) service_pool.cc:225] Handling call
0507 08:05:59.057530 (+1181587us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b540014fad4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:59.057687 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1065497472}
I20240507 08:05:59.059074 10332 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:57.953408 10337 rpcz_store.cc:276] 0507 08:05:56.942219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:56.942342 (+ 123us) service_pool.cc:225] Handling call
0507 08:05:57.001948 (+ 59606us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:57.002172 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:57.559259 (+557087us) log.cc:842] Serialized 14 byte log entry
0507 08:05:57.945946 (+386687us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:57.945983 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:57.946010 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:57.946051 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:57.946124 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23340,"prepare.queue_time_us":501576,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":10449,"raft.queue_time_us":25855,"thread_start_us":909963,"threads_started":2}
I20240507 08:05:59.030613 11696 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:59.073199 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:59.073679 10191 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:59.078078 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:59.079572 10341 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:05:59.080049 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:59.080767 10337 raft_consensus.cc:2396] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240507 08:05:59.080822 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:59.030587 11918 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.084830 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:59.086912 10341 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:59.091775 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:59.092628 10336 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:59.102336 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:05:59.104807 11948 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:59.108103 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:59.110875 10155 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.090785 11778 consensus_queue.cc:1046] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20240507 08:05:59.111999 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1267) took 1133 ms. Trace:
I20240507 08:05:59.077942 11696 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.115773 11918 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:59.116417 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1270) took 1109 ms. Trace:
I20240507 08:05:59.116575 11696 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:05:59.116811 10264 rpcz_store.cc:276] 0507 08:05:58.006735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.825134 (+818399us) service_pool.cc:225] Handling call
0507 08:05:58.830024 (+ 4890us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.830401 (+ 377us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.830444 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.830965 (+ 521us) log.cc:842] Serialized 26 byte log entry
0507 08:05:58.939110 (+108145us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:58.973403 (+ 34293us) log.cc:842] Serialized 14 byte log entry
0507 08:05:58.973662 (+ 259us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.973763 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.116237 (+142474us) raft_consensus.cc:1671] finished
0507 08:05:59.116255 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.116366 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":253,"prepare.queue_time_us":35703,"raft.queue_time_us":14182,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":11281,"spinlock_wait_cycles":83414144,"thread_start_us":107707,"threads_started":1,"wal-append.queue_time_us":232395}
I20240507 08:05:59.118999 10264 raft_consensus.cc:1261] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:05:59.124575 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:59.112349 10267 rpcz_store.cc:276] 0507 08:05:57.978311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.622157 (+643846us) service_pool.cc:225] Handling call
0507 08:05:58.622497 (+ 340us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.628873 (+ 6376us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.628917 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.643924 (+ 15007us) log.cc:842] Serialized 26 byte log entry
0507 08:05:58.852161 (+208237us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:58.906920 (+ 54759us) log.cc:842] Serialized 14 byte log entry
0507 08:05:58.907167 (+ 247us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.913700 (+ 6533us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.111805 (+198105us) raft_consensus.cc:1671] finished
0507 08:05:59.111835 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.111946 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3746,"prepare.queue_time_us":41412,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":3996,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":9370368,"thread_start_us":160033,"threads_started":1,"wal-append.queue_time_us":242706}
I20240507 08:05:59.138065 10337 raft_consensus.cc:2379] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:05:59.142968 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet e9adf1535c974b2abf910f3c1c537627 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:05:59.144176 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:59.144414 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:05:59.150030 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:05:59.151477 10304 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.158725 10233 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:59.149487 11960 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:59.167428 10304 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:59.176013 10304 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:59.168092 11696 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240507 08:05:59.178946 10304 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.187242 10333 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:05:59.187927 11824 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:59.173393 11960 raft_consensus.cc:686] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.174629 11788 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:59.189327 10304 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:59.180881 11936 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:05:59.202368 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47576.3us
I20240507 08:05:59.198743 11788 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:05:59.201419 11756 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:59.202188 11824 consensus_queue.cc:1046] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20240507 08:05:59.215874 10192 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:05:59.204309 10304 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.219410 10304 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:59.207157 11654 raft_consensus.cc:3032] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:59.210712 11788 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:59.230954 11696 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:05:59.232158 11712 raft_consensus.cc:3032] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:05:59.232478 11712 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.216121 11846 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:59.213687 11936 raft_consensus.cc:686] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.240312 11936 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.226812 10304 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:59.245134 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 975) took 1396 ms. Trace:
I20240507 08:05:59.245545 10265 rpcz_store.cc:276] 0507 08:05:57.848266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:57.993255 (+144989us) service_pool.cc:225] Handling call
0507 08:05:58.018922 (+ 25667us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.143208 (+124286us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.143235 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.143607 (+ 372us) log.cc:842] Serialized 26 byte log entry
0507 08:05:58.305650 (+162043us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:58.356099 (+ 50449us) log.cc:842] Serialized 14 byte log entry
0507 08:05:58.374784 (+ 18685us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.654086 (+279302us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b180034eb20. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8decab06 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089
0507 08:05:59.239298 (+585212us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540027db54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:59.239349 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.239395 (+ 46us) raft_consensus.cc:1671] finished
0507 08:05:59.239408 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.245078 (+ 5670us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17789,"prepare.queue_time_us":83860,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":44239,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":4909,"spinlock_wait_cycles":318637568,"thread_start_us":153666,"threads_started":1,"wal-append.queue_time_us":257876}
I20240507 08:05:59.210932 11756 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:05:59.250111 11813 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:59.256661 11813 raft_consensus.cc:686] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.257613 11813 consensus_queue.cc:237] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:59.263512 11982 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.891s user 0.002s sys 0.000s
W20240507 08:05:59.270058 10262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1272) took 1251 ms (client timeout 1634 ms). Trace:
W20240507 08:05:59.270536 10262 rpcz_store.cc:269] 0507 08:05:58.018350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.229922 (+211572us) service_pool.cc:225] Handling call
0507 08:05:59.269883 (+1039961us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b5400358c54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:59.270001 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76775168}
I20240507 08:05:59.271929 10262 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:59.273185 10262 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240507 08:05:59.273634 11849 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:59.275807 11852 raft_consensus.cc:3032] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:59.276214 11852 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:59.277029 11852 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.277940 11937 raft_consensus.cc:3032] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:59.278285 11852 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:59.278470 11937 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.279618 11937 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.221297 11824 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:59.221108 11837 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:05:59.283123 10266 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:59.281101 11937 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:59.284881 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:05:59.226485 11654 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:59.265156 11756 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:59.291749 10262 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:59.210440 11960 consensus_queue.cc:237] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.294126 10265 raft_consensus.cc:1226] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:59.233255 11712 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.292861 11837 raft_consensus.cc:686] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.315018 11712 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:59.325150 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 976) took 1476 ms. Trace:
I20240507 08:05:59.325583 10263 rpcz_store.cc:276] 0507 08:05:57.848506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.006961 (+158455us) service_pool.cc:225] Handling call
0507 08:05:58.031153 (+ 24192us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.042274 (+ 11121us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.042302 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.042684 (+ 382us) log.cc:842] Serialized 26 byte log entry
0507 08:05:58.221993 (+179309us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:58.242130 (+ 20137us) log.cc:842] Serialized 14 byte log entry
0507 08:05:58.242321 (+ 191us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:58.242380 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.262868 (+1020488us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:05:59.322318 (+ 59450us) raft_consensus.cc:1671] finished
0507 08:05:59.324955 (+ 2637us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.325093 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30534,"prepare.queue_time_us":20017,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":87188,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":19955,"spinlock_wait_cycles":7415040,"thread_start_us":67644,"threads_started":1,"wal-append.queue_time_us":172336}
I20240507 08:05:59.335074 11601 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.336128 11601 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:59.337635 11879 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:59.338599 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1269) took 1343 ms. Trace:
I20240507 08:05:59.338960 10261 rpcz_store.cc:276] 0507 08:05:57.995623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.644305 (+648682us) service_pool.cc:225] Handling call
0507 08:05:58.646460 (+ 2155us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.849961 (+203501us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.338300 (+488339us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400117954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:59.338533 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449885056}
I20240507 08:05:59.305840 11654 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.348620 11914 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:05:59.349031 11914 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:59.340987 11837 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.348624 11654 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:59.347011 11848 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.354327 11848 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:59.360005 10336 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:59.371163 11788 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:59.387223 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 359690us
I20240507 08:05:59.388427 10231 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:59.387367 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 325070us
I20240507 08:05:59.389954 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1345) took 1276 ms. Trace:
I20240507 08:05:59.390309 10189 rpcz_store.cc:276] 0507 08:05:58.113527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.113667 (+ 140us) service_pool.cc:225] Handling call
0507 08:05:58.113967 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:58.509610 (+395643us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.389750 (+880140us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54003b4a54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:59.389900 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":793897344}
I20240507 08:05:59.390872 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 980) took 1247 ms. Trace:
I20240507 08:05:59.372771 11756 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:59.391201 10259 rpcz_store.cc:276] 0507 08:05:58.143941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.930526 (+786585us) service_pool.cc:225] Handling call
0507 08:05:58.943120 (+ 12594us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.972715 (+ 29595us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.972748 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.977311 (+ 4563us) log.cc:842] Serialized 26 byte log entry
0507 08:05:59.080493 (+103182us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:59.099491 (+ 18998us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.257967 (+158476us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b480002ff58. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:05:59.390510 (+132543us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.390588 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.390629 (+ 41us) raft_consensus.cc:1671] finished
0507 08:05:59.390704 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.390817 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":319825,"prepare.queue_time_us":13346,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":43,"raft.queue_time_us":30787,"raft.run_cpu_time_us":3374,"raft.run_wall_time_us":282815,"spinlock_wait_cycles":6189824,"thread_start_us":102763,"threads_started":1,"wal-append.queue_time_us":143826}
W20240507 08:05:59.407708 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 249136us
I20240507 08:05:59.391558 10232 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:59.410713 10304 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:05:59.411351 10183 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:05:59.411770 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.417507 10188 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:05:59.422322 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:59.423327 10231 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.392112 11958 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:59.467584 10259 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 1. Current term is 2. Ops: []
I20240507 08:05:59.468983 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1266) took 1496 ms. Trace:
I20240507 08:05:59.471287 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 928) took 1714 ms. Trace:
I20240507 08:05:59.471668 10334 rpcz_store.cc:276] 0507 08:05:57.757247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.492966 (+735719us) service_pool.cc:225] Handling call
0507 08:05:58.850359 (+357393us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:58.852360 (+ 2001us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.471124 (+618764us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400339ad0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:59.471237 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":821521664}
I20240507 08:05:59.473995 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1347) took 1359 ms. Trace:
W20240507 08:05:59.474723 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 50869.9us
I20240507 08:05:59.469363 10258 rpcz_store.cc:276] 0507 08:05:57.973076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.603358 (+630282us) service_pool.cc:225] Handling call
0507 08:05:58.603706 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:58.604134 (+ 428us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:58.604175 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:58.623148 (+ 18973us) log.cc:842] Serialized 26 byte log entry
0507 08:05:59.251697 (+628549us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b180030b7a0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:05:59.334383 (+ 82686us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:59.346658 (+ 12275us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.422586 (+ 75928us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.422650 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.422690 (+ 40us) raft_consensus.cc:1671] finished
0507 08:05:59.468799 (+ 46109us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.468932 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":75094,"prepare.queue_time_us":11824,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":112,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":20218,"spinlock_wait_cycles":6000512,"thread_start_us":82510,"threads_started":1,"wal-append.queue_time_us":697066}
I20240507 08:05:59.413208 11962 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:59.471797 11960 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:59.418320 11949 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:59.474375 10184 rpcz_store.cc:276] 0507 08:05:58.114104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.114202 (+ 98us) service_pool.cc:225] Handling call
0507 08:05:58.121276 (+ 7074us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:58.259947 (+138671us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:05:58.302084 (+ 42137us) log.cc:842] Serialized 14 byte log entry
0507 08:05:58.627188 (+325104us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:58.627240 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:58.627346 (+ 106us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.473804 (+846458us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54000474d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:05:59.473845 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.473943 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16942,"prepare.queue_time_us":35996,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":1818,"raft.queue_time_us":21366,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":3539,"spinlock_wait_cycles":318394240,"thread_start_us":337739,"threads_started":2,"wal-append.queue_time_us":396631}
I20240507 08:05:59.418169 11778 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240507 08:05:59.483592 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 92126.9us
I20240507 08:05:59.420318 11814 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:05:59.486158 10183 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 1357) took 1361 ms (client timeout 1644 ms). Trace:
W20240507 08:05:59.506955 10183 rpcz_store.cc:269] 0507 08:05:58.124996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.210458 (+ 85462us) service_pool.cc:225] Handling call
0507 08:05:59.485941 (+1275483us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000479d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:59.486084 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2572067584}
I20240507 08:05:59.510566 10334 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 2. Current term is 3. Ops: []
I20240507 08:05:59.475019 11788 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:59.479148 11959 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:59.517746 11814 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.466442 11813 consensus_queue.cc:1046] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20240507 08:05:59.487926 11962 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:05:59.519281 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 438721us
W20240507 08:05:59.521965 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 98772.1us
I20240507 08:05:59.505813 11976 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:59.514420 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:59.529119 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:59.530257 10189 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
W20240507 08:05:59.525470 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 414788us
I20240507 08:05:59.522547 11853 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:59.531234 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:39708 (request call id 254) took 1561 ms. Trace:
I20240507 08:05:59.535332 10334 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:59.568176 10185 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:59.560454 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:05:59.564554 10187 raft_consensus.cc:1261] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:59.555861 10306 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.558909 10183 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:59.560143 10155 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.533362 10231 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.551776 11756 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:05:59.576892 11846 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:05:59.577423 11846 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.578369 11846 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.579953 11958 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:59.580952 11958 consensus_queue.cc:1225] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240507 08:05:59.567085 11853 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:05:59.581550 11958 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:05:59.550493 11814 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.583289 11958 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:05:59.536373 11696 consensus_queue.cc:1046] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20240507 08:05:59.542582 11972 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:59.591331 11972 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.593278 11972 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:59.549960 11978 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:59.547029 12024 consensus_queue.cc:1046] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20240507 08:05:59.535521 11813 raft_consensus.cc:3032] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Stepping down as leader of term 1
I20240507 08:05:59.556612 11736 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:59.582193 11853 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:59.595793 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:05:59.648603 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 52956.3us
I20240507 08:05:59.598991 11959 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:05:59.649251 10155 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.644656 11894 raft_consensus.cc:3032] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:05:59.650950 10155 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:59.651155 11894 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:05:59.532135 11941 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:05:59.652014 11941 raft_consensus.cc:686] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:05:59.613413 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1045) took 1729 ms. Trace:
I20240507 08:05:59.652693 10332 rpcz_store.cc:276] 0507 08:05:57.883776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.602230 (+718454us) service_pool.cc:225] Handling call
0507 08:05:59.058911 (+456681us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:59.061066 (+ 2155us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.613146 (+552080us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b540014fad0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:59.613310 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":988934528}
I20240507 08:05:59.615633 10334 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20240507 08:05:59.594098 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 60875.4us
I20240507 08:05:59.654940 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.656287 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.611498 11741 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:05:59.569061 11924 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:59.567265 10338 rpcz_store.cc:276] 0507 08:05:57.959293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.695106 (+735813us) service_pool.cc:225] Handling call
0507 08:05:59.520025 (+824919us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b540014ebd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dacd30 00007fcf90e3017a 00007fcf95d6da0e 00007fcf8f460817 00007fcf8f4607a2 00007fcf8f4606d1 00007fcf8f46064c 00007fcf8f45f902 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:05:59.520411 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401874816}
I20240507 08:05:59.630606 10183 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
W20240507 08:05:59.663659 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 172418us
I20240507 08:05:59.637068 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:05:59.640017 10335 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:05:59.640141 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:05:59.643251 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 47417us
I20240507 08:05:59.666167 10341 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:05:59.644636 11738 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:05:59.667909 10232 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:05:59.618021 11813 raft_consensus.cc:729] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:05:59.651302 10340 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:05:59.605105 10307 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.657562 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.652400 11601 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:05:59.670580 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 987) took 1385 ms. Trace:
I20240507 08:05:59.670457 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:05:59.671252 10261 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:59.652027 11750 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:59.674396 10231 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:05:59.665645 11654 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:05:59.664945 11959 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:05:59.663908 11741 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:59.662489 11958 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:05:59.673323 11736 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:05:59.686939 11941 consensus_queue.cc:237] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.664731 11852 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:05:59.685456 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 124080us
I20240507 08:05:59.711362 10307 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:05:59.666448 11924 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.690276 11741 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:05:59.678088 11915 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:05:59.689348 11830 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:05:59.685699 11756 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:05:59.665478 11899 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20240507 08:05:59.677744 11866 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20240507 08:05:59.687536 11813 consensus_queue.cc:260] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.708509 10156 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:05:59.715826 10261 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:05:59.672384 10260 rpcz_store.cc:276] 0507 08:05:58.285613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.028836 (+743223us) service_pool.cc:225] Handling call
0507 08:05:59.045862 (+ 17026us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:59.387102 (+341240us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:05:59.389289 (+ 2187us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.517845 (+128556us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:59.518274 (+ 429us) log.cc:842] Serialized 26 byte log entry
0507 08:05:59.518411 (+ 137us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:59.518621 (+ 210us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.670183 (+151562us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.670252 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.670299 (+ 47us) raft_consensus.cc:1671] finished
0507 08:05:59.670418 (+ 119us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.670529 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":383896,"prepare.queue_time_us":16841,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":81,"raft.queue_time_us":386417,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":14950,"spinlock_wait_cycles":13940480,"thread_start_us":236443,"threads_started":2,"wal-append.queue_time_us":199410}
I20240507 08:05:59.673516 11601 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:59.690004 11957 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:05:59.690191 11952 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:59.652264 11894 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:05:59.677879 11750 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:05:59.681612 11654 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:05:59.756004 10333 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:05:59.685755 11976 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20240507 08:05:59.690292 11861 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.077s
I20240507 08:05:59.772549 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1278) took 1418 ms. Trace:
I20240507 08:05:59.755026 11829 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:05:59.708173 11972 consensus_queue.cc:1035] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20240507 08:05:59.776696 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 68270.8us
W20240507 08:05:59.808475 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 122524us
I20240507 08:05:59.759549 11736 raft_consensus.cc:686] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:05:59.780467 10334 raft_consensus.cc:1261] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:05:59.766294 11601 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:59.830328 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:05:59.844815 10336 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:05:59.710450 11738 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.054s
I20240507 08:05:59.726651 11852 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.775761 11825 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:59.836709 10264 rpcz_store.cc:276] 0507 08:05:58.354585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.120483 (+765898us) service_pool.cc:225] Handling call
0507 08:05:59.120768 (+ 285us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:05:59.414234 (+293466us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:59.414264 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:05:59.414749 (+ 485us) log.cc:842] Serialized 52 byte log entry
0507 08:05:59.667637 (+252888us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:59.692138 (+ 24501us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.709538 (+ 17400us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.772178 (+ 62640us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.772269 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:05:59.772322 (+ 53us) raft_consensus.cc:1671] finished
0507 08:05:59.772341 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.772490 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":445773,"prepare.queue_time_us":44832,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":413455,"raft.run_cpu_time_us":416,"raft.run_wall_time_us":46645,"thread_start_us":184517,"threads_started":1,"wal-append.queue_time_us":272557}
I20240507 08:05:59.783290 11833 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:05:59.839958 10341 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:59.838176 11813 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:05:59.783668 11750 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:59.783515 11654 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:05:59.830530 10340 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:05:59.862339 10340 raft_consensus.cc:595] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:05:59.862691 10340 raft_consensus.cc:603] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:05:59.848841 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1063) took 1264 ms. Trace:
I20240507 08:05:59.863354 10339 rpcz_store.cc:276] 0507 08:05:58.584071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.848133 (+1264062us) service_pool.cc:225] Handling call
0507 08:05:59.848404 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:59.848557 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:59.848579 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:05:59.848620 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:05:59.848646 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.848695 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:05:59.848793 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:05:59.841076 11936 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.025s sys 0.030s
I20240507 08:05:59.861452 10260 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:05:59.832924 11848 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240507 08:05:59.778250 11952 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:05:59.870098 11936 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.025s sys 0.031s
I20240507 08:05:59.858644 11833 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:05:59.917327 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 72810.8us
I20240507 08:05:59.890689 12037 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:05:59.861511 11825 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:05:59.911334 11833 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:05:59.874780 11738 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20240507 08:05:59.890255 11866 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.079s
W20240507 08:06:00.043934 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 168137us
I20240507 08:06:00.049196 10336 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
W20240507 08:06:00.041237 12039 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
W20240507 08:06:00.055729 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 207473us
I20240507 08:06:00.065439 10264 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240507 08:06:00.056017 12034 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.366s user 0.001s sys 0.005s
I20240507 08:06:00.050830 11936 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.087153 11936 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.069149 11957 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240507 08:06:00.091079 12049 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.091543 12049 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.091104 11811 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.094440 11811 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.096778 11811 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.090508 11741 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.060542 11976 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.098227 11976 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.100062 11976 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:05:59.951738 11736 consensus_queue.cc:237] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.107576 10186 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
W20240507 08:06:00.059484 12043 log.cc:924] Time spent T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.198s user 0.000s sys 0.005s
I20240507 08:06:00.075189 10184 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.079039 11962 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.080215 10260 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:00.082620 11925 raft_consensus.cc:3032] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:00.083076 11959 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:00.129829 11849 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.059316 11915 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:00.130321 11849 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.130656 11915 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.132122 11849 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.132594 11915 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.068142 11860 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:00.133471 11860 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.134913 10260 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:00.135156 11914 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.135859 11860 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.136112 11914 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:00.061954 11899 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.136441 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.137092 10264 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:06:00.136934 11899 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.112274 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.138273 10191 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:06:00.139547 11899 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.140555 10260 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.141398 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1280) took 1737 ms. Trace:
I20240507 08:06:00.141768 10267 rpcz_store.cc:276] 0507 08:05:58.403552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.127525 (+723973us) service_pool.cc:225] Handling call
0507 08:05:59.127907 (+ 382us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:59.406929 (+279022us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:59.406957 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:05:59.407329 (+ 372us) log.cc:842] Serialized 26 byte log entry
0507 08:05:59.617601 (+210272us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:05:59.757120 (+139519us) log.cc:842] Serialized 14 byte log entry
0507 08:05:59.830343 (+ 73223us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:05:59.830406 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:00.141156 (+310750us) raft_consensus.cc:1671] finished
0507 08:06:00.141212 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:00.141342 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":352679,"prepare.queue_time_us":47843,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":355943,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":3230,"spinlock_wait_cycles":9019136,"thread_start_us":233782,"threads_started":2,"wal-append.queue_time_us":252580}
I20240507 08:06:00.144866 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.145416 10267 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:00.146581 10260 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: [4.5-4.5]
W20240507 08:06:00.147406 12045 log.cc:924] Time spent T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20240507 08:06:00.148087 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 988) took 1862 ms. Trace:
I20240507 08:06:00.148474 10266 rpcz_store.cc:276] 0507 08:05:58.285877 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.029890 (+744013us) service_pool.cc:225] Handling call
0507 08:05:59.282890 (+253000us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:59.284345 (+ 1455us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:00.147922 (+863577us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b5400358c50. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:00.148035 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":581358976}
I20240507 08:06:00.150979 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.152777 10260 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 1.
I20240507 08:06:00.156139 10184 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:00.160212 11741 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.151113 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1381) took 1502 ms. Trace:
I20240507 08:05:59.841145 10940 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:00.120004 10336 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:00.170477 10232 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.120615 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.122614 10340 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240507 08:06:00.128288 11924 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:00.253473 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 83099.3us
I20240507 08:06:00.128423 10192 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:00.111603 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.163223 10188 rpcz_store.cc:276] 0507 08:05:58.648604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.648995 (+ 391us) service_pool.cc:225] Handling call
0507 08:05:58.666176 (+ 17181us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:59.417909 (+751733us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:00.150915 (+733006us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b5400260f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:00.151061 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1728124288}
I20240507 08:06:00.163928 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.164460 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:00.156985 10089 catalog_manager.cc:5484] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:00.257135 10089 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 3 to 5, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New cstate: current_term: 2 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:00.168982 10258 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:00.093448 12049 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.058403 11738 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.065s
I20240507 08:06:00.251749 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.171358 10185 raft_consensus.cc:1203] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3] Dedup: 3.3->[]
I20240507 08:06:00.267895 10185 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240507 08:06:00.268534 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.269634 10266 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:00.252624 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.245910 10341 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:06:00.253484 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.252588 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:00.172873 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 69224.1us
I20240507 08:06:00.059846 11853 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.048265 12037 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.272953 10262 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:00.279778 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.283217 10191 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:06:00.115379 11976 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:00.118155 11925 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:00.277333 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.292712 10155 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.115202 12051 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:00.298775 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 134075us
I20240507 08:06:00.271476 10089 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:00.285064 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.300014 10184 raft_consensus.cc:2379] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:00.300448 10089 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 2 to 4, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129), now has a pending config: VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129), VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130), VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131), VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:00.273335 10260 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
W20240507 08:06:00.271260 12040 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.432s user 0.001s sys 0.001s
I20240507 08:06:00.268502 11958 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:00.276132 11741 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.281831 11959 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.280894 11957 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.291250 11861 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.290990 11756 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.304189 11941 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:00.358932 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.300674 11852 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.315058 11899 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:00.284770 11866 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20240507 08:06:00.326577 11958 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.378254 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.335268 11744 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.284597 11973 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20240507 08:06:00.303141 11976 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.340902 11936 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:00.387281 10337 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:06:00.368065 11941 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.358265 11824 raft_consensus.cc:3032] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:00.404043 10340 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.356087 11745 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:00.368526 11899 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.341949 12049 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20240507 08:06:00.416618 10340 raft_consensus.cc:1261] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240507 08:06:00.389458 11958 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.389083 11825 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:00.310530 12051 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:00.424816 12055 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20240507 08:06:00.350849 11741 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.423965 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.354956 11957 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.462404 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.361320 11861 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.354388 11654 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:00.366858 11756 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.465459 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.466042 10340 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
W20240507 08:06:00.475951 10186 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 94) took 1607 ms (client timeout 1689 ms). Trace:
I20240507 08:06:00.492605 10337 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
W20240507 08:06:00.492661 10186 rpcz_store.cc:269] 0507 08:05:58.868462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.912733 (+ 44271us) service_pool.cc:225] Handling call
0507 08:06:00.475755 (+1563022us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001b3854. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:00.475883 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2698935040}
W20240507 08:06:00.416244 12041 log.cc:924] Time spent T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.361s user 0.000s sys 0.001s
I20240507 08:06:00.415463 11952 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:00.351393 11959 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:00.466269 12046 log.cc:924] Time spent T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.414s user 0.000s sys 0.001s
I20240507 08:06:00.507413 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.504688 10184 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:06:00.501538 10333 raft_consensus.cc:2396] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240507 08:06:00.414539 11941 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:00.418268 11962 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:00.505452 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.515380 10332 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:00.468974 11894 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.510803 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:00.419131 11899 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.417420 11824 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:00.479470 11750 consensus_meta.cc:287] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.602s user 0.010s sys 0.000s
I20240507 08:06:00.477241 11957 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.401301 11744 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.522275 10337 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:00.391182 11925 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.414446 11745 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.563748 11696 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:00.565905 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:00.566565 10337 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:00.569818 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.570645 10262 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:06:00.525944 11853 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.570896 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:00.557669 11813 consensus_meta.cc:287] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.691s user 0.007s sys 0.003s
I20240507 08:06:00.585887 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:00.587668 10260 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:00.595870 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.597544 10156 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:00.598436 10156 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:00.591738 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.599834 12049 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:00.600639 11754 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:00.601140 12049 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:00.565765 11925 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.550938 11936 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20240507 08:06:00.523236 11962 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:00.601560 12049 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.615101 12047 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:00.548575 11750 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:00.572286 11830 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.713s user 0.012s sys 0.000s
I20240507 08:06:00.624078 10337 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
W20240507 08:06:00.644021 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 305269us
I20240507 08:06:00.589465 12053 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:00.563782 10908 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:00.586481 11829 consensus_queue.cc:1046] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20240507 08:06:00.660864 10260 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:00.610065 11860 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:00.648909 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:00.654189 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 351419us
I20240507 08:06:00.623580 11811 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.641605 12049 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:00.636232 12037 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.638968 12047 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.673846 12053 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.704701 10260 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.702418 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.706490 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.717373 10266 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:06:00.675047 11861 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.717406 10188 raft_consensus.cc:2396] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:06:00.721530 10258 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.706019 10262 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.712172 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.634750 11973 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.069s
I20240507 08:06:00.678916 11745 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.679489 11860 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:00.594254 11824 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:00.701750 11833 consensus_meta.cc:287] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.695s user 0.011s sys 0.000s
I20240507 08:06:00.727829 10261 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:06:00.697818 11768 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:00.713467 11741 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.695641 11976 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.695957 11924 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:00.733660 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.697667 11696 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:00.697278 11754 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:00.766491 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 50639.1us
I20240507 08:06:00.728281 11756 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:00.720816 12053 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.701925 11654 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:00.777711 10265 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:00.696882 11811 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.780723 11824 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:00.698603 11825 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
W20240507 08:06:00.769498 11601 consensus_meta.cc:287] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.912s user 0.008s sys 0.004s
I20240507 08:06:00.788197 12060 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:00.698968 11992 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20240507 08:06:00.732203 11959 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.790799 11745 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:00.725674 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 162020us
W20240507 08:06:00.787256 11914 consensus_meta.cc:287] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.651s user 0.004s sys 0.008s
I20240507 08:06:00.717729 12047 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:00.844044 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:00.844388 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.730262 11920 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20240507 08:06:00.778467 11696 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:00.776562 11960 consensus_queue.cc:1159] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "59382e6647c549d89d83c70233c0667e" responder_term: 4 status { last_received { term: 4 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 4 index: 3 } }
I20240507 08:06:00.793303 12069 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:00.733697 11860 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:00.788895 12051 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:00.789484 11924 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:00.783082 11768 raft_consensus.cc:686] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:00.814635 11811 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.791656 11744 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.777768 11833 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.848275 10191 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:00.788077 11754 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.845302 10265 raft_consensus.cc:2396] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20240507 08:06:00.793715 11957 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20240507 08:06:00.807893 11829 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:00.856377 10263 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:00.812640 11601 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.819170 11745 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.888856 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.876823 11750 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.892839 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.894325 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.894862 10190 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:06:00.895213 12053 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.895678 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.896111 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.896644 11914 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:00.897061 11914 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.897310 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.897432 11914 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.897794 12053 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:00.898577 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.898893 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.899272 10340 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:00.899933 10190 raft_consensus.cc:1261] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:00.899961 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.901052 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.901134 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.902338 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.902696 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.903975 11976 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:00.904188 10340 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:06:00.889341 11696 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.906951 11696 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:00.816164 12060 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.909096 12060 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.914016 10183 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
W20240507 08:06:00.887796 12057 log.cc:924] Time spent T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.225s user 0.000s sys 0.001s
I20240507 08:06:00.914906 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.915462 10260 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:00.915709 10258 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:00.905150 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.916183 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:00.913554 10333 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:00.917119 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.917498 10333 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:06:00.918125 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.883517 11813 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:00.919709 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.920569 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.921144 10340 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:00.921882 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:00.923918 10155 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:00.925561 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.926163 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:00.926227 10191 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
W20240507 08:06:00.913594 12065 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20240507 08:06:00.928319 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:00.928350 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:00.930240 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.940361 10155 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.931699 11924 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:00.932842 12066 log.cc:924] Time spent T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20240507 08:06:00.933107 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.932859 11754 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.929534 11962 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:00.928570 10232 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.865136 12069 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.922611 11976 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:00.889118 11829 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:00.958593 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:00.958990 12069 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:00.929852 12060 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:00.962636 10942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55926 (request call id 3) took 1109 ms. Trace:
I20240507 08:06:00.963124 10259 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20240507 08:06:00.925494 11813 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.946411 11853 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:00.941864 11848 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:00.970007 10938 raft_consensus.cc:1799] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Handling vote request from an unknown peer 59382e6647c549d89d83c70233c0667e
W20240507 08:06:00.970218 12068 log.cc:924] Time spent T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20240507 08:06:00.952589 11736 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:00.975014 11829 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:00.944917 11879 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:00.927892 11973 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:00.980906 10942 rpcz_store.cc:276] 0507 08:05:59.890376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.050739 (+160363us) service_pool.cc:225] Handling call
0507 08:06:00.962226 (+911487us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b54006d15d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dacd30 00007fcf90e3017a 00007fcf95d6da0e 00007fcf8f460817 00007fcf8f4607a2 00007fcf8f4606d1 00007fcf8f46064c 00007fcf8f45f902 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:00.962565 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1329118464}
I20240507 08:06:00.988477 10260 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:00.989485 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:00.976243 12044 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:00.994865 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.001889 10258 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
W20240507 08:06:00.983718 10940 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.608s user 0.009s sys 0.001s
W20240507 08:06:00.986231 12081 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20240507 08:06:01.000737 10155 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:00.986117 11853 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.989375 11848 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:01.005679 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:00.944798 11768 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:00.968863 11976 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.009812 10265 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:00.996668 11736 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:00.998232 11879 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:00.986239 12060 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:00.989648 11962 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240507 08:06:01.022261 11879 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:06:01.027428 12082 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20240507 08:06:01.028206 12060 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.008630 11853 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.029542 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 990) took 1255 ms. Trace:
I20240507 08:06:01.018126 10938 raft_consensus.cc:2396] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240507 08:06:01.030683 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.010282 11848 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:01.012018 11736 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.015832 11973 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:01.057660 11973 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:01.058094 11899 consensus_meta.cc:287] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.509s user 0.005s sys 0.005s
I20240507 08:06:01.058604 11899 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.058539 11973 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:01.030561 10335 rpcz_store.cc:276] 0507 08:05:59.774216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.774704 (+ 488us) service_pool.cc:225] Handling call
0507 08:05:59.775087 (+ 383us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:06:00.030375 (+255288us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:00.030413 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:06:00.054639 (+ 24226us) log.cc:842] Serialized 52 byte log entry
0507 08:06:00.386176 (+331537us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:00.463948 (+ 77772us) log.cc:842] Serialized 14 byte log entry
0507 08:06:00.767333 (+303385us) log.cc:842] Serialized 14 byte log entry
0507 08:06:00.807897 (+ 40564us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:00.808002 (+ 105us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:01.029340 (+221338us) raft_consensus.cc:1671] finished
0507 08:06:01.029359 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:01.029486 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123636,"prepare.queue_time_us":469009,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":81,"raft.queue_time_us":49945,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":10383,"spinlock_wait_cycles":18391808,"thread_start_us":739825,"threads_started":3,"wal-append.queue_time_us":527691}
I20240507 08:06:01.040398 10265 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:01.063521 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.064190 10186 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:01.045063 11976 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.045709 11745 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.033069 11601 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.053120 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.036514 11952 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:01.054818 12078 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20240507 08:06:01.015869 11833 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.086185 11952 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:01.071206 11958 consensus_meta.cc:287] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.564s user 0.008s sys 0.001s
I20240507 08:06:01.061286 11813 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:01.061388 11899 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.057200 11914 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:01.098124 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 97511.6us
I20240507 08:06:01.101939 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.056067 12047 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:01.103031 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:01.097733 11941 consensus_meta.cc:287] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.574s user 0.006s sys 0.005s
I20240507 08:06:01.116035 10266 raft_consensus.cc:2379] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:01.118721 11879 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.119776 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.139017 10260 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 1 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.086859 11853 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.157048 11853 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:01.128647 12087 log.cc:924] Time spent T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20240507 08:06:01.129631 11925 consensus_meta.cc:287] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.511s user 0.007s sys 0.003s
I20240507 08:06:01.132648 10265 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.136083 12047 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:01.135962 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:01.159410 10185 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:01.137938 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:01.125137 12061 log.cc:924] Time spent T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.424s user 0.000s sys 0.002s
I20240507 08:06:01.136667 11941 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.148396 11879 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.148849 10934 raft_consensus.cc:2354] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:01.150758 11860 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.201035 10187 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:01.202062 11860 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.202142 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.202798 10262 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.203083 10189 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:01.202131 11941 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.212929 10262 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:01.215303 11915 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:01.151326 10260 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:01.215791 11915 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.216945 12069 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:01.217499 12069 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.154489 11736 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.136993 12052 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:01.163342 10158 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.170199 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.219026 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 996) took 1099 ms. Trace:
I20240507 08:06:01.241480 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.242122 10259 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.220923 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.242898 10260 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.240731 12052 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:01.244125 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 80949.7us
I20240507 08:06:01.244441 12052 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.246081 11736 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.247506 12069 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.241408 10339 rpcz_store.cc:276] 0507 08:06:00.120004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.120114 (+ 110us) service_pool.cc:225] Handling call
0507 08:06:00.120405 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:00.261290 (+140885us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:00.261319 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:00.277071 (+ 15752us) log.cc:842] Serialized 26 byte log entry
0507 08:06:00.490947 (+213876us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:00.491001 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:00.491098 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:01.124467 (+633369us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:01.218745 (+ 94278us) raft_consensus.cc:1671] finished
0507 08:06:01.218776 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:01.218964 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":140546,"raft.queue_time_us":148023,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":7685,"spinlock_wait_cycles":31669632,"thread_start_us":209874,"threads_started":1,"wal-append.queue_time_us":343232}
I20240507 08:06:01.250464 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.250478 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.228205 10265 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:01.231226 10155 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.095121 11958 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.217526 11811 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:01.175144 11753 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:01.254354 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:01.255218 10155 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.253664 10267 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:01.255561 10184 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.258199 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.253711 11833 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.258760 11958 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.258955 12094 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:01.259516 11860 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:01.259680 11972 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:01.260031 11860 raft_consensus.cc:686] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.260174 11972 raft_consensus.cc:686] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.260950 11972 consensus_queue.cc:237] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.222764 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.260895 11860 consensus_queue.cc:237] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.259531 12094 raft_consensus.cc:686] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.265553 12094 consensus_queue.cc:237] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.260733 10260 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
W20240507 08:06:01.274714 12093 log.cc:924] Time spent T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20240507 08:06:01.275698 11753 raft_consensus.cc:686] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.277009 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.276947 11753 consensus_queue.cc:237] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.278147 11925 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:01.278731 11925 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.279667 11925 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.262400 11811 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.290242 10259 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:01.291998 10184 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:01.302595 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1417) took 1027 ms. Trace:
I20240507 08:06:01.302901 10185 rpcz_store.cc:276] 0507 08:06:00.275004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.275179 (+ 175us) service_pool.cc:225] Handling call
0507 08:06:00.280676 (+ 5497us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:01.159752 (+879076us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.302423 (+142671us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b54001b4754. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:01.302551 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2020523392}
I20240507 08:06:01.304006 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1408) took 1433 ms. Trace:
I20240507 08:06:01.304296 10183 rpcz_store.cc:276] 0507 08:05:59.870834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.905558 (+ 34724us) service_pool.cc:225] Handling call
0507 08:06:00.041897 (+136339us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:00.914387 (+872490us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.303855 (+389468us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b54001bf8d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:01.303965 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2005937536}
W20240507 08:06:01.306447 11830 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s user 0.007s sys 0.007s
W20240507 08:06:01.307032 11830 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.008s sys 0.007s
I20240507 08:06:01.312705 10155 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.312649 10232 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:01.313645 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 58935.3us
I20240507 08:06:01.319384 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:01.282146 11915 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.319990 10185 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.323495 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:01.325618 12047 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:01.326104 12047 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:01.326821 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:01.327545 10183 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.326913 12047 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:01.337838 10183 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:01.338785 10232 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.319068 11736 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:01.345778 10185 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:01.334530 11754 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:01.352581 11848 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.339526 11768 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:01.342716 12069 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:01.342484 11914 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.362749 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.363240 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.344717 10940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 110) took 1653 ms. Trace:
I20240507 08:06:01.365373 10940 rpcz_store.cc:276] 0507 08:05:59.691260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.709043 (+ 17783us) service_pool.cc:225] Handling call
0507 08:05:59.775079 (+ 66036us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:05:59.832021 (+ 56942us) raft_consensus.cc:1520] Early marking committed up to index 5
0507 08:06:01.008681 (+1176660us) log.cc:842] Serialized 14 byte log entry
0507 08:06:01.296121 (+287440us) log.cc:842] Serialized 14 byte log entry
0507 08:06:01.320528 (+ 24407us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:01.331431 (+ 10903us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:01.334637 (+ 3206us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.339294 (+ 4657us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:01.342155 (+ 2861us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29765,"prepare.queue_time_us":52228,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":36,"raft.queue_time_us":65818,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":19693,"thread_start_us":158019,"threads_started":2,"wal-append.queue_time_us":263040}
I20240507 08:06:01.348337 11736 raft_consensus.cc:686] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:01.373286 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.374181 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:01.374630 11985 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:01.376497 11825 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:01.377008 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 498734us
I20240507 08:06:01.378285 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.357602 11958 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:01.379354 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 496852us
I20240507 08:06:01.387637 10335 raft_consensus.cc:1799] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Handling vote request from an unknown peer 59382e6647c549d89d83c70233c0667e
I20240507 08:06:01.380875 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.388190 10335 raft_consensus.cc:2379] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:01.388840 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.388919 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.389441 10339 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.381107 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.390007 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.390282 10341 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.381556 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.365379 10185 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:01.386245 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.391527 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.392585 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:01.393285 10185 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:01.394134 10334 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:01.372066 12051 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:01.371533 12069 raft_consensus.cc:686] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:01.395449 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.372094 11754 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.383523 11992 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20240507 08:06:01.401929 10232 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.402779 10185 raft_consensus.cc:2379] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 3: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:01.406246 10156 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:01.409400 10183 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:01.408921 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.409529 11915 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:01.412456 11915 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.412863 11920 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:01.414299 11879 raft_consensus.cc:2741] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20240507 08:06:01.414397 11915 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:01.417258 12049 consensus_meta.cc:287] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.702s user 0.002s sys 0.008s
I20240507 08:06:01.386341 11811 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.410877 10341 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:01.426448 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.427297 10258 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:01.364744 11768 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:01.400671 11849 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240507 08:06:01.407493 12051 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.435928 12051 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:01.438817 10158 leader_election.cc:398] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Vote denied by peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849) with higher term. Message: Invalid argument: T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:01.394555 11985 raft_consensus.cc:686] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:01.442395 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:01.444833 12058 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.875s user 0.005s sys 0.001s
I20240507 08:06:01.447765 10339 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
W20240507 08:06:01.450512 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 63520.8us
W20240507 08:06:01.450662 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 62755us
I20240507 08:06:01.409039 11848 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.452800 12069 consensus_queue.cc:237] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.426751 11959 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:01.455232 11985 consensus_queue.cc:237] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.456382 11754 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.392446 11825 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.458591 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.426440 12049 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.394775 11958 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.457067 11952 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.472800 10304 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:01.461125 11959 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:01.463620 11788 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:01.408954 12052 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.483563 11788 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:01.487160 11825 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:01.488356 11788 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:01.489084 11958 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.490541 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:01.491240 10934 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:01.491691 12049 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.505157 10304 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.506244 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.506858 10259 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 7
I20240507 08:06:01.481050 11959 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:01.508327 10156 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.512081 10306 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:01.513533 10231 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:01.514930 10259 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:01.515341 10258 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:01.519342 10156 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:01.522727 12059 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.899s user 0.003s sys 0.003s
I20240507 08:06:01.523643 12052 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.480623 12063 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20240507 08:06:01.524158 11837 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.524756 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:01.478616 12085 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20240507 08:06:01.525907 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.526892 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.525224 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.527408 10262 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:01.528200 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.529233 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.530615 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 991) took 1754 ms. Trace:
I20240507 08:06:01.517647 11972 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.530973 10338 rpcz_store.cc:276] 0507 08:05:59.779021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:59.779166 (+ 145us) service_pool.cc:225] Handling call
0507 08:05:59.834380 (+ 55214us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:05:59.834579 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:05:59.834602 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:00.140549 (+305947us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b2c0015a408. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8f809f11 00007fcf8f819542 00007fcf90e62fc1 00007fcf90e277da 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782
0507 08:06:00.140874 (+ 325us) log.cc:842] Serialized 26 byte log entry
0507 08:06:00.455592 (+314718us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:00.511966 (+ 56374us) log.cc:842] Serialized 14 byte log entry
0507 08:06:00.523092 (+ 11126us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:00.523163 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:01.085996 (+562833us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b1800401fe0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd2e4 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:06:01.511056 (+425060us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:01.530425 (+ 19369us) raft_consensus.cc:1671] finished
0507 08:06:01.530448 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:01.530559 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35102,"prepare.queue_time_us":56506,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":11275,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6883,"spinlock_wait_cycles":45141376,"thread_start_us":203765,"threads_started":1,"wal-append.queue_time_us":420507}
I20240507 08:06:01.533710 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1001) took 1394 ms. Trace:
I20240507 08:06:01.535112 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.534067 10264 rpcz_store.cc:276] 0507 08:06:00.139565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.142547 (+ 2982us) service_pool.cc:225] Handling call
0507 08:06:00.142816 (+ 269us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:00.143115 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 2
0507 08:06:00.143384 (+ 269us) log.cc:842] Serialized 14 byte log entry
0507 08:06:00.398464 (+255080us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:00.407572 (+ 9108us) log.cc:842] Serialized 26 byte log entry
0507 08:06:00.407838 (+ 266us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:00.408279 (+ 441us) log.cc:842] Serialized 14 byte log entry
0507 08:06:00.464280 (+ 56001us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:00.464347 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:01.101355 (+637008us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b18001f8640. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd2e4 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:06:01.444297 (+342942us) spinlock_profiling.cc:243] Waited 6.36 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:01.533500 (+ 89203us) raft_consensus.cc:1671] finished
0507 08:06:01.533523 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:01.533659 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70222,"prepare.queue_time_us":2920,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":75405,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":10794,"spinlock_wait_cycles":22951552,"thread_start_us":240106,"threads_started":1,"wal-append.queue_time_us":365709}
I20240507 08:06:01.467025 11736 consensus_queue.cc:237] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.537715 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.539414 10189 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:01.539969 10189 raft_consensus.cc:595] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.516332 12071 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20240507 08:06:01.541236 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:01.542411 12107 log.cc:924] Time spent T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20240507 08:06:01.518354 11788 consensus_queue.cc:1046] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20240507 08:06:01.546000 10231 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:01.547183 10231 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:01.511282 11825 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:01.549108 10267 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:01.550124 11914 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:01.550483 12051 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:01.562163 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 997) took 1428 ms. Trace:
I20240507 08:06:01.562860 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.550588 11811 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.511059 11813 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.551498 12060 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.566299 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.567260 10188 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:01.567831 10188 raft_consensus.cc:595] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.556027 10261 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20240507 08:06:01.568694 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.562539 10263 rpcz_store.cc:276] 0507 08:06:00.133952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.139074 (+ 5122us) service_pool.cc:225] Handling call
0507 08:06:00.139311 (+ 237us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:00.885830 (+746519us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.561958 (+676128us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b540023bed4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:01.562109 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1625232640}
I20240507 08:06:01.548314 10231 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:01.578466 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 44306.5us
I20240507 08:06:01.579756 10185 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240507 08:06:01.563463 10338 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:06:01.595012 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7d05e73be847e99048092c9570faed with delete_type TABLET_DATA_TOMBSTONED (TS 59382e6647c549d89d83c70233c0667e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:55810
W20240507 08:06:01.595309 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 47083.3us
I20240507 08:06:01.582190 11972 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:01.597016 10231 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:01.584105 11958 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
I20240507 08:06:01.597710 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1421) took 1023 ms. Trace:
I20240507 08:06:01.578500 12051 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.557106 11825 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:01.581224 12112 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20240507 08:06:01.599468 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:01.607077 10263 raft_consensus.cc:2379] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:01.613951 10338 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:01.616279 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.599723 10089 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 69) took 2616 ms. Trace:
I20240507 08:06:01.561637 11837 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:01.598068 10187 rpcz_store.cc:276] 0507 08:06:00.574801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.574942 (+ 141us) service_pool.cc:225] Handling call
0507 08:06:00.575197 (+ 255us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:01.201465 (+626268us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.597474 (+396009us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b54000479d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:01.597649 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1439402752}
I20240507 08:06:01.567720 11899 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.600337 10231 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:01.603129 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.603722 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:01.605278 10261 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:01.788193 10339 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20240507 08:06:01.788190 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 188023us
I20240507 08:06:01.791096 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:01.606884 11825 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.566325 11937 consensus_queue.cc:1046] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
W20240507 08:06:01.766697 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 148820us
I20240507 08:06:01.777513 10089 rpcz_store.cc:276] 0507 08:05:58.976554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.976688 (+ 134us) service_pool.cc:225] Handling call
0507 08:06:01.593310 (+2616622us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:00.615143 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:00.615624 (+ 481us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:00.615661 (+ 37us) write_op.cc:435] Acquired schema lock
0507 08:06:00.615678 (+ 17us) tablet.cc:660] Decoding operations
0507 08:06:00.616247 (+ 569us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:00.616304 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:00.616329 (+ 25us) tablet.cc:683] Acquiring locks for 4 operations
0507 08:06:00.616794 (+ 465us) tablet.cc:699] Row locks acquired
0507 08:06:00.616814 (+ 20us) write_op.cc:260] PREPARE: finished
0507 08:06:00.616937 (+ 123us) write_op.cc:270] Start()
0507 08:06:00.617038 (+ 101us) write_op.cc:276] Timestamp: P: 1715069160616895 usec, L: 0
0507 08:06:00.617056 (+ 18us) op_driver.cc:331] REPLICATION: starting
0507 08:06:00.618874 (+ 1818us) log.cc:842] Serialized 1999 byte log entry
0507 08:06:01.247869 (+628995us) op_driver.cc:444] REPLICATION: finished
0507 08:06:01.372829 (+124960us) write_op.cc:301] APPLY: starting
0507 08:06:01.379284 (+ 6455us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:01.402147 (+ 22863us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:01.411484 (+ 9337us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:01.435219 (+ 23735us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:01.435394 (+ 175us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0507 08:06:01.435434 (+ 40us) write_op.cc:312] APPLY: finished
0507 08:06:01.435643 (+ 209us) log.cc:842] Serialized 40 byte log entry
0507 08:06:01.435797 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:01.435902 (+ 105us) write_op.cc:454] Released schema lock
0507 08:06:01.435948 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:01.436234 (+ 286us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:01.436264 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":110670,"apply.run_cpu_time_us":2762,"apply.run_wall_time_us":71344,"num_ops":4,"prepare.queue_time_us":287045,"prepare.run_cpu_time_us":24871,"prepare.run_wall_time_us":263127,"raft.queue_time_us":219262,"raft.run_cpu_time_us":33321,"raft.run_wall_time_us":93635,"replication_time_us":630724,"thread_start_us":556826,"threads_started":4,"wal-append.queue_time_us":299034}]]}
I20240507 08:06:01.769801 11751 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:01.792598 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:01.771193 10908 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.773818 12085 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:01.798501 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.799166 10334 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:01.799932 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.585031 11962 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:01.778259 10338 raft_consensus.cc:595] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:01.803205 10338 raft_consensus.cc:603] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:01.600639 11811 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.800889 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.814582 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.801093 10334 raft_consensus.cc:2396] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240507 08:06:01.811858 10231 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:01.818557 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.820014 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.820960 10337 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:01.801221 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.801561 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.804533 10263 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:06:01.808838 10264 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:01.812031 10188 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
W20240507 08:06:01.812963 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 252709us
I20240507 08:06:01.824010 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:01.822232 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:01.825304 10335 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:01.826411 10304 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.776314 11914 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:01.793320 11833 consensus_meta.cc:287] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.534s user 0.012s sys 0.000s
W20240507 08:06:01.803797 12115 log.cc:924] Time spent T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.205s user 0.005s sys 0.000s
W20240507 08:06:01.929863 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 158816us
I20240507 08:06:01.948122 11925 raft_consensus.cc:979] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:01.802590 12063 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:01.787904 11957 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:01.949672 11753 raft_consensus.cc:979] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:01.827554 11861 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
W20240507 08:06:01.952035 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 125715us
W20240507 08:06:01.952232 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 142868us
I20240507 08:06:01.994521 10304 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:01.944368 12085 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.955885 10332 raft_consensus.cc:1261] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:01.799760 11985 raft_consensus.cc:2741] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20240507 08:06:02.016076 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 192779us
I20240507 08:06:01.796243 11768 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:01.796674 11972 raft_consensus.cc:686] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:01.827713 11848 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.176s
I20240507 08:06:01.816622 11899 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
I20240507 08:06:01.988197 10335 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:01.793087 10907 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.966833 11869 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:01.802688 11959 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:01.972918 11958 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:01.971647 12071 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:06:01.817528 11915 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.022554 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 1025) took 1107 ms. Trace:
I20240507 08:06:02.018883 11849 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:01.980556 11833 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:02.076692 11849 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.076529 11899 raft_consensus.cc:3032] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.077875 11899 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:02.078595 11899 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:02.035534 10306 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:01.972684 11914 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
W20240507 08:06:02.080426 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 47128.9us
I20240507 08:06:02.081640 11914 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.082127 10188 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:02.082422 12053 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:06:02.082556 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 476833us
I20240507 08:06:02.084074 10188 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:02.084610 10188 raft_consensus.cc:595] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.084642 11768 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:02.085163 10188 raft_consensus.cc:603] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:02.085290 11768 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:02.085786 11768 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:02.086620 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:02.088258 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:02.088836 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:02.094429 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.095209 10338 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:02.098146 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.098390 11972 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.104990 11959 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.106057 12123 tablet_replica.cc:330] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: stopping tablet replica
I20240507 08:06:02.020539 11811 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:02.030997 12051 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.046058 11696 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:02.046355 11813 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:02.126008 10264 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:01.985944 12063 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.074900 11866 raft_consensus.cc:1067] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:02.075775 11848 raft_consensus.cc:1067] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:02.076498 10266 rpcz_store.cc:276] 0507 08:06:00.915261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.928031 (+ 12770us) service_pool.cc:225] Handling call
0507 08:06:02.022356 (+1094325us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54002bf554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:02.022492 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":408179840}
I20240507 08:06:02.111353 10260 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20240507 08:06:02.120865 10338 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:02.122363 10259 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:02.079185 11849 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.124274 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1106) took 1495 ms. Trace:
I20240507 08:06:02.144762 10333 rpcz_store.cc:276] 0507 08:06:00.628490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:00.921832 (+293342us) service_pool.cc:225] Handling call
0507 08:06:00.922039 (+ 207us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:01.400968 (+478929us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:01.400996 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:01.401379 (+ 383us) log.cc:842] Serialized 26 byte log entry
0507 08:06:01.516017 (+114638us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:06:01.516933 (+ 916us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.517012 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:02.124069 (+607057us) raft_consensus.cc:1671] finished
0507 08:06:02.124091 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:02.124223 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":478650,"raft.queue_time_us":479744,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":5846,"thread_start_us":104062,"threads_started":1,"wal-append.queue_time_us":188313}
I20240507 08:06:02.125501 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.128755 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:02.029218 11830 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:02.028863 12049 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:02.108496 12121 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20240507 08:06:02.135275 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.148591 10263 raft_consensus.cc:2379] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:02.126266 11813 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:02.179158 11813 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:02.112316 11811 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:02.181852 11811 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:02.185961 11712 raft_consensus.cc:3032] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.186347 11712 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:02.187053 11712 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.187695 11869 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.188014 11712 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.148838 12053 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.188995 12053 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:02.189716 12053 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.190821 12053 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.149389 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:02.156003 12049 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.164372 11830 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:02.165331 11837 raft_consensus.cc:3032] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:02.174860 12085 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.176607 12063 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.130991 11914 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:02.201426 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 228553us
I20240507 08:06:02.130705 11825 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:02.213835 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 219439us
I20240507 08:06:02.219539 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.224047 12085 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:02.225257 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.226114 10188 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:02.227154 10304 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.227355 10184 raft_consensus.cc:1261] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:02.199016 11962 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:02.229434 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 212439us
I20240507 08:06:02.211772 11837 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:02.205860 11957 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20240507 08:06:02.246615 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.252207 10339 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 4
I20240507 08:06:02.254943 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.255638 10334 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:02.259750 12136 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:02.261075 12136 raft_consensus.cc:3032] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:02.261490 12136 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:02.262434 12136 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.263178 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.263924 12136 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:02.264037 10263 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:02.265380 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.266343 10265 raft_consensus.cc:2354] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:06:02.267117 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.273008 11846 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:02.273635 11846 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.274587 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:02.276423 11846 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.277099 11846 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:02.277990 11846 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:02.278570 11846 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:02.279222 10304 leader_election.cc:398] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:06:02.225507 11914 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:02.293944 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.279942 10304 leader_election.cc:401] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:02.232142 12129 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240507 08:06:02.297770 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 486159us
I20240507 08:06:02.304895 10231 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.311514 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.322536 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.323684 10231 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.329465 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:02.330996 10231 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.251803 12070 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:02.325177 11736 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:02.354292 10263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:02.354836 10263 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:02.340957 10266 raft_consensus.cc:1261] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:02.355155 10263 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.356640 10185 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:06:02.357080 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.358860 10231 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:02.331576 10334 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:02.333448 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.360654 10334 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:02.280793 11959 consensus_queue.cc:1035] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20240507 08:06:02.348292 10262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:02.320580 11924 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:02.324759 11957 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:02.352063 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.362254 10262 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:02.371366 10262 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.362965 10339 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:02.373862 10339 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:02.384769 10339 raft_consensus.cc:1261] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:02.386765 10339 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:02.323385 11866 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20240507 08:06:02.358389 11936 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:02.347281 12140 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.400534 10262 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.442673 10260 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:02.437386 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:02.337191 11972 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20240507 08:06:02.308749 11962 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.445559 10263 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.349658 11601 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20240507 08:06:02.367589 11915 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.402668 11736 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.296890 11837 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.390108 12123 raft_consensus.cc:2229] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:02.379006 11744 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:02.378582 12071 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4.
I20240507 08:06:02.448606 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:02.475044 10190 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.475371 10190 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:02.448736 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.445999 11825 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.391592 11830 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.452174 10262 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.404624 12049 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.446974 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:02.454576 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 95808.6us
W20240507 08:06:02.454882 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 82209.9us
I20240507 08:06:02.463716 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.368909 12051 consensus_queue.cc:1046] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.078s
I20240507 08:06:02.450428 10263 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.375468 11924 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.406522 12069 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20240507 08:06:02.356003 12070 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.447172 11741 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:02.476030 10190 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.494333 10190 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.489140 10306 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.446491 11936 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.490664 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.492236 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:02.499354 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:02.500349 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:02.501046 10191 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.501456 10191 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:02.502081 10262 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:02.503546 10262 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20240507 08:06:02.478428 12085 consensus_queue.cc:1159] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20240507 08:06:02.464267 12132 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.502353 10191 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.515177 10339 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:02.517203 10263 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20240507 08:06:02.518539 10263 raft_consensus.cc:1261] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:02.520301 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.521044 10337 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.521139 10263 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20240507 08:06:02.510654 11825 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:02.530580 10337 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:02.500646 11992 raft_consensus.cc:3032] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.487759 11768 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:02.540947 10190 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:02.554365 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.554689 10231 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:02.557140 10232 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:02.557634 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:02.559583 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:02.559830 10306 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.560937 10156 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.500278 11741 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.571056 10156 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.530401 12129 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.532755 12053 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:02.514102 12131 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20240507 08:06:02.555552 11992 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:02.576248 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.583995 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.584312 10156 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:02.586293 10337 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.563189 11973 consensus_queue.cc:1035] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20240507 08:06:02.566219 11848 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.571076 12140 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:02.534314 11853 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20240507 08:06:02.566534 11825 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:02.591837 12151 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:02.565855 11814 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:02.572712 12138 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240507 08:06:02.600730 12144 log.cc:924] Time spent T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20240507 08:06:02.570132 11866 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20240507 08:06:02.558701 11829 consensus_queue.cc:1046] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20240507 08:06:02.577361 12123 raft_consensus.cc:2258] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:02.566756 11754 consensus_queue.cc:1046] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20240507 08:06:02.562203 12135 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20240507 08:06:02.646016 10337 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:02.588677 12053 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.570214 12049 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20240507 08:06:02.570617 11811 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.603091 11601 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240507 08:06:02.678939 10231 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.649s (SENT) [suppressed 30 similar messages]
W20240507 08:06:02.592485 12145 log.cc:924] Time spent T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20240507 08:06:02.689711 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.630789 12140 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:02.666981 12044 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20240507 08:06:02.723379 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:02.697304 11712 consensus_meta.cc:287] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.509s user 0.009s sys 0.001s
I20240507 08:06:02.725407 10337 raft_consensus.cc:2379] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:02.690225 12151 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:02.695278 11829 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:02.729653 10336 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:02.672032 11814 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.710263 12070 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.728226 12140 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.736248 12053 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:02.743822 12147 log.cc:924] Time spent T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20240507 08:06:02.732400 12050 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20240507 08:06:02.743041 12067 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:02.743942 12150 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
W20240507 08:06:02.759584 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 175440us
W20240507 08:06:02.772632 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 93754.7us
I20240507 08:06:02.727277 11936 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.532608 11952 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20240507 08:06:02.731572 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.801770 10337 raft_consensus.cc:2396] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20240507 08:06:02.792448 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 235433us
W20240507 08:06:02.793355 12141 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
W20240507 08:06:02.793757 12143 log.cc:924] Time spent T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20240507 08:06:02.794142 12067 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.796283 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:02.732555 12146 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20240507 08:06:02.729101 11924 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.685767 11992 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.733754 11829 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:02.800771 11768 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:02.822432 12157 log.cc:924] Time spent T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240507 08:06:02.824528 10267 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
W20240507 08:06:02.808454 11813 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.629s user 0.010s sys 0.000s
I20240507 08:06:02.837875 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:02.835577 12142 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20240507 08:06:02.840658 11829 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.842455 11814 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.842859 12151 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.843096 11813 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:02.877130 10341 raft_consensus.cc:2354] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6.
I20240507 08:06:02.846429 12158 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:02.878094 12158 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.878818 11813 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:02.879395 11813 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:02.879972 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:02.879823 11813 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.880553 10934 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:02.880599 12158 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.848060 12067 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.848380 11811 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:02.849270 11846 consensus_meta.cc:287] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.570s user 0.008s sys 0.003s
I20240507 08:06:02.882499 11846 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.883098 11813 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.884940 11846 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:02.885600 12161 log.cc:924] Time spent T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20240507 08:06:02.885800 10332 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:02.847110 11741 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:02.851754 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.887662 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:02.889031 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:02.890689 10338 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:02.852207 12123 ts_tablet_manager.cc:1875] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:02.901810 12159 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:02.902443 12159 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:02.911993 12159 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:02.915122 10934 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:03.211010 11741 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:03.194000 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 386807us
I20240507 08:06:03.198514 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1059) took 1074 ms. Trace:
I20240507 08:06:03.201027 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.213764 10261 rpcz_store.cc:276] 0507 08:06:02.123716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.123851 (+ 135us) service_pool.cc:225] Handling call
0507 08:06:02.144519 (+ 20668us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:02.154093 (+ 9574us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:02.154136 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:02.154615 (+ 479us) log.cc:842] Serialized 26 byte log entry
0507 08:06:02.294372 (+139757us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:02.462355 (+167983us) log.cc:842] Serialized 14 byte log entry
0507 08:06:02.462612 (+ 257us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:02.462743 (+ 131us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:03.197949 (+735206us) raft_consensus.cc:1671] finished
0507 08:06:03.198002 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.198322 (+ 320us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9147,"prepare.queue_time_us":203028,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":10859,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":9547,"thread_start_us":305073,"threads_started":2,"wal-append.queue_time_us":183912}
I20240507 08:06:02.855922 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:03.218093 10267 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 4
I20240507 08:06:03.204072 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.224257 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.226981 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.227306 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:03.204681 10306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.624s user 0.009s sys 0.017s
W20240507 08:06:03.229835 10306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.649s user 0.010s sys 0.018s
I20240507 08:06:03.193092 10231 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:03.231300 10192 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:03.233021 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:02.809513 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.213048 11601 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.237473 11601 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.240525 10339 raft_consensus.cc:1261] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:03.244560 10304 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:03.245227 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47396 (request call id 124) took 1110 ms. Trace:
I20240507 08:06:03.212335 11741 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.251037 11601 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.251466 10334 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 6. Ops: []
I20240507 08:06:03.254405 10339 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:03.245743 10341 rpcz_store.cc:276] 0507 08:06:02.134814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.134987 (+ 173us) service_pool.cc:225] Handling call
0507 08:06:03.244769 (+1109782us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b540004d3d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:03.245018 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1666341760}
I20240507 08:06:03.264398 10156 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:03.271337 10336 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1013) took 1473 ms (client timeout 1649 ms). Trace:
W20240507 08:06:03.271888 10336 rpcz_store.cc:269] 0507 08:06:01.797910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:01.801778 (+ 3868us) service_pool.cc:225] Handling call
0507 08:06:02.293775 (+491997us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:06:03.270974 (+977199us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b540004ced4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:03.271150 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1007084800}
I20240507 08:06:03.273315 11696 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.274662 10332 raft_consensus.cc:1261] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:03.276293 11696 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.272092 10339 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:02.859524 11829 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:02.885566 11846 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:02.934897 12130 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:03.005779 12133 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:03.064806 12153 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.464s user 0.000s sys 0.010s
I20240507 08:06:03.369830 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1129) took 1119 ms. Trace:
I20240507 08:06:03.370283 10335 rpcz_store.cc:276] 0507 08:06:02.250453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.250629 (+ 176us) service_pool.cc:225] Handling call
0507 08:06:02.250948 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:02.251166 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 5
0507 08:06:02.280909 (+ 29743us) log.cc:842] Serialized 14 byte log entry
0507 08:06:03.268409 (+987500us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b1800275e60. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae4ab 00007fcf90e31ada
0507 08:06:03.369254 (+100845us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:03.369320 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:03.369376 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.369448 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.369643 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52678,"prepare.queue_time_us":28965,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"spinlock_wait_cycles":9231616,"thread_start_us":21879,"threads_started":1,"wal-append.queue_time_us":1043609}
I20240507 08:06:03.373703 12156 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:03.373984 11846 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.286906 10266 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:03.260390 12172 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:03.285308 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.294452 11696 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:03.319409 10264 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:03.320123 11813 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:03.379768 11846 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:03.379696 11813 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.323738 10155 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:03.338892 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:03.347280 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 67560.8us
I20240507 08:06:03.347986 11936 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:03.351141 10907 leader_election.cc:398] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6.
W20240507 08:06:03.345353 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 78992.6us
I20240507 08:06:03.364804 12130 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:03.369246 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 70906.3us
I20240507 08:06:03.378134 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1330) took 1114 ms. Trace:
I20240507 08:06:03.378466 12172 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.386286 12138 consensus_queue.cc:1046] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20240507 08:06:03.386971 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 70304us
I20240507 08:06:03.382014 11936 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:03.394228 10192 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:03.384657 10259 rpcz_store.cc:276] 0507 08:06:02.261016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.261175 (+ 159us) service_pool.cc:225] Handling call
0507 08:06:02.261510 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:02.511468 (+249958us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:02.511496 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:02.511535 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:02.511562 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.374928 (+863366us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b5400116554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.374983 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.375195 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":574633216}
I20240507 08:06:03.404166 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:03.405891 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:03.406023 12133 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.406870 10264 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:03.408576 12133 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:03.409180 11962 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:03.410002 11962 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.410414 11899 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:03.396519 11936 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:03.419144 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.426098 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:03.421555 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:03.429997 11866 raft_consensus.cc:3032] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
W20240507 08:06:03.430616 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 238024us
I20240507 08:06:03.417295 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.435820 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.414906 11741 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:03.442384 12164 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240507 08:06:03.443675 12115 log.cc:924] Time spent T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20240507 08:06:03.448650 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:03.449398 12049 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:03.450408 10231 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:03.452734 12172 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:03.457607 12170 log.cc:924] Time spent T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20240507 08:06:03.464386 12063 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:03.464833 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:03.465751 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 180122us
I20240507 08:06:03.464462 10189 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:03.436357 11866 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:03.472682 12164 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:03.473167 12164 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:03.473556 12164 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:03.417035 12130 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.474627 12130 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:03.475097 11962 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:03.475967 10263 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:03.482108 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1448) took 1153 ms. Trace:
I20240507 08:06:03.482707 10231 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:03.483919 11741 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:03.482553 10184 rpcz_store.cc:276] 0507 08:06:02.328430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.330472 (+ 2042us) service_pool.cc:225] Handling call
0507 08:06:02.350499 (+ 20027us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:02.505559 (+155060us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:06:02.531021 (+ 25462us) log.cc:842] Serialized 14 byte log entry
0507 08:06:02.570873 (+ 39852us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:02.570924 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:02.570965 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.481537 (+910572us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400261954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.481622 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.481831 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1301,"prepare.queue_time_us":26771,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":2793,"raft.queue_time_us":1476,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":2229,"spinlock_wait_cycles":356110720,"thread_start_us":38150,"threads_started":1,"wal-append.queue_time_us":177280}
I20240507 08:06:03.486016 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1134) took 1132 ms. Trace:
I20240507 08:06:03.486479 10334 rpcz_store.cc:276] 0507 08:06:02.354121 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.362250 (+ 8129us) service_pool.cc:225] Handling call
0507 08:06:03.250848 (+888598us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:03.264934 (+ 14086us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.485805 (+220871us) spinlock_profiling.cc:243] Waited 884 ms on lock 0x7b540004d3d0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:03.485964 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2033244032}
W20240507 08:06:03.488497 10338 consensus_meta.cc:287] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.597s user 0.012s sys 0.005s
I20240507 08:06:03.473349 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.488456 10334 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:03.491930 11736 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:03.491988 10335 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:03.473416 11866 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.490257 11813 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:03.494153 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.498828 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:03.504887 10267 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:06:03.510886 12151 consensus_queue.cc:1035] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240507 08:06:03.512671 12151 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.512998 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.514084 12172 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:03.515167 10259 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:03.509336 10338 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
W20240507 08:06:03.513195 12120 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 1.477s user 0.018s sys 0.004s
I20240507 08:06:03.520929 10264 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:03.527074 10907 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:03.529975 12162 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:03.513273 12151 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:03.554427 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 150690us
I20240507 08:06:03.557242 10265 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:03.558568 12151 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.561553 10155 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:03.573843 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1462) took 1162 ms. Trace:
I20240507 08:06:03.574410 10190 rpcz_store.cc:276] 0507 08:06:02.411206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.578104 (+166898us) service_pool.cc:225] Handling call
0507 08:06:02.578432 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:02.597479 (+ 19047us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:06:02.625288 (+ 27809us) log.cc:842] Serialized 14 byte log entry
0507 08:06:02.758395 (+133107us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:02.758461 (+ 66us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:02.758518 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.573196 (+814678us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b54002dafd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.573296 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.573633 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":28187,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":507,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"spinlock_wait_cycles":43357568,"thread_start_us":132055,"threads_started":1,"wal-append.queue_time_us":161220}
I20240507 08:06:03.581220 10191 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:03.583220 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1314) took 2061 ms. Trace:
I20240507 08:06:03.583598 10258 rpcz_store.cc:276] 0507 08:06:01.521513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:01.521644 (+ 131us) service_pool.cc:225] Handling call
0507 08:06:01.521944 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:01.552694 (+ 30750us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:01.552725 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:01.553112 (+ 387us) log.cc:842] Serialized 26 byte log entry
0507 08:06:01.785320 (+232208us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:01.785354 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:01.785425 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:03.512112 (+1726687us) spinlock_profiling.cc:243] Waited 70.5 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:03.582767 (+ 70655us) raft_consensus.cc:1671] finished
0507 08:06:03.582792 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.583035 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28501,"raft.queue_time_us":24152,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":15432,"spinlock_wait_cycles":162135936,"thread_start_us":231930,"threads_started":1,"wal-append.queue_time_us":441339}
W20240507 08:06:03.586144 12149 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 1.016s user 0.006s sys 0.012s
I20240507 08:06:03.587828 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 116) took 1274 ms. Trace:
I20240507 08:06:03.588181 10183 rpcz_store.cc:276] 0507 08:06:02.313794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.313987 (+ 193us) service_pool.cc:225] Handling call
0507 08:06:02.315207 (+ 1220us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:02.327964 (+ 12757us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:02.327993 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:02.334683 (+ 6690us) log.cc:842] Serialized 26 byte log entry
0507 08:06:02.489886 (+155203us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:02.489933 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:02.490001 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:03.585329 (+1095328us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:03.587421 (+ 2092us) raft_consensus.cc:1671] finished
0507 08:06:03.587441 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.587671 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23677,"raft.queue_time_us":21111,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":6684,"spinlock_wait_cycles":6991104,"thread_start_us":154810,"threads_started":1,"wal-append.queue_time_us":220626}
I20240507 08:06:03.578230 10304 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:03.604599 12064 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.606917 12064 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.606845 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1069) took 1233 ms. Trace:
I20240507 08:06:03.611413 10337 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:03.610253 10260 rpcz_store.cc:276] 0507 08:06:02.361422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.365051 (+ 3629us) service_pool.cc:225] Handling call
0507 08:06:02.365382 (+ 331us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:02.443256 (+ 77874us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.551920 (+1108664us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b54002adfd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.594877 (+ 42957us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177243392}
W20240507 08:06:03.619978 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 63658.6us
W20240507 08:06:03.625294 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47167.9us
I20240507 08:06:03.625021 11852 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
W20240507 08:06:03.627458 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 66047.7us
I20240507 08:06:03.632575 12179 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:03.633930 10262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1081) took 1068 ms. Trace:
I20240507 08:06:03.634389 10262 rpcz_store.cc:276] 0507 08:06:02.566232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.215429 (+649197us) service_pool.cc:225] Handling call
0507 08:06:03.216135 (+ 706us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:03.332375 (+116240us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:03.332406 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:03.364501 (+ 32095us) log.cc:842] Serialized 26 byte log entry
0507 08:06:03.435198 (+ 70697us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:03.435236 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.435304 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:03.633670 (+198366us) raft_consensus.cc:1671] finished
0507 08:06:03.633692 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.633867 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":109751,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":287,"thread_start_us":160615,"threads_started":2,"wal-append.queue_time_us":256204}
I20240507 08:06:03.612803 12162 raft_consensus.cc:686] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:03.640343 12183 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.641063 10187 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20240507 08:06:03.641455 10187 pending_rounds.cc:76] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 6
I20240507 08:06:03.641016 12183 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.641705 10187 pending_rounds.cc:97] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20240507 08:06:03.641965 10187 raft_consensus.cc:2866] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:03.642107 11972 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:03.642591 11972 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:03.645072 11972 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:03.646433 11914 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:03.647207 12085 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:03.647811 12085 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:03.650313 11936 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.654060 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1026) took 1004 ms. Trace:
I20240507 08:06:03.666144 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.674317 12064 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:03.674453 10188 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1072) took 1546 ms (client timeout 2057 ms). Trace:
I20240507 08:06:03.676631 10337 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:03.627275 11852 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:03.658478 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.678333 10341 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:03.633431 12179 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.679780 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:03.682541 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:03.682924 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 60290.7us
I20240507 08:06:03.683813 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.684435 10338 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
W20240507 08:06:03.676404 10188 rpcz_store.cc:269] 0507 08:06:02.127955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.128468 (+ 513us) service_pool.cc:225] Handling call
0507 08:06:03.671550 (+1543082us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b540033f4d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:03.674275 (+ 2725us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178667264}
I20240507 08:06:03.687868 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.688841 12186 consensus_queue.cc:1035] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240507 08:06:03.658532 10333 rpcz_store.cc:276] 0507 08:06:02.486002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.235793 (+749791us) service_pool.cc:225] Handling call
0507 08:06:03.236330 (+ 537us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:03.236924 (+ 594us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:06:03.306425 (+ 69501us) log.cc:842] Serialized 14 byte log entry
0507 08:06:03.489769 (+183344us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:03.489818 (+ 49us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:03.489861 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.489925 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.490036 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15347,"prepare.queue_time_us":118738,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":19566,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":114,"thread_start_us":205634,"threads_started":2,"wal-append.queue_time_us":302079}
W20240507 08:06:03.638998 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 85565.8us
I20240507 08:06:03.701047 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.701339 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.701758 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.680442 11936 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:03.709434 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.694815 12130 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.714397 10231 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:03.719226 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:03.719350 11830 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:03.720539 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.722205 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.723325 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:03.692656 12064 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.726369 12064 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.713826 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:03.732462 10191 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:03.706454 12137 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20240507 08:06:03.738822 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1079) took 1286 ms (client timeout 1622 ms). Trace:
W20240507 08:06:03.739187 10191 rpcz_store.cc:269] 0507 08:06:02.452915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.487691 (+ 34776us) service_pool.cc:225] Handling call
0507 08:06:03.580804 (+1093113us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b58008d0eb0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90da31a3 00007fcf90e06d0a 00007fcf90e1e5ff 00007fcf90e1fd9e 00007fcf90e2b478 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c
0507 08:06:03.738767 (+157963us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2451976064}
I20240507 08:06:03.730379 12064 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:03.741291 10341 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:03.736439 12130 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.745797 10341 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240507 08:06:03.745682 12160 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.031s
I20240507 08:06:03.747291 10231 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:03.730129 12183 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:03.752658 12070 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:03.752678 12060 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:03.754115 12186 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:03.753600 12151 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:03.756042 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1016) took 1957 ms. Trace:
I20240507 08:06:03.743876 11852 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.734287 12164 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.769259 10340 rpcz_store.cc:276] 0507 08:06:01.798697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:01.828203 (+ 29506us) service_pool.cc:225] Handling call
0507 08:06:01.828609 (+ 406us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:01.993468 (+164859us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:01.993497 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:02.017263 (+ 23766us) log.cc:842] Serialized 26 byte log entry
0507 08:06:03.242839 (+1225576us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b18000aa4a0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:06:03.395898 (+153059us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:03.447912 (+ 52014us) log.cc:842] Serialized 14 byte log entry
0507 08:06:03.463979 (+ 16067us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.464105 (+ 126us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:03.488379 (+ 24274us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b4800180258. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:06:03.755743 (+267364us) raft_consensus.cc:1671] finished
0507 08:06:03.755762 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.755884 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":193902,"prepare.queue_time_us":11220,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":74,"raft.queue_time_us":191116,"raft.run_cpu_time_us":3407,"raft.run_wall_time_us":29613,"spinlock_wait_cycles":55428096,"thread_start_us":86883,"threads_started":1,"wal-append.queue_time_us":1693972}
I20240507 08:06:03.771209 10183 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:03.772152 12164 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.772635 10340 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:03.774976 12151 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.775975 12123 ts_tablet_manager.cc:1888] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240507 08:06:03.763713 10341 pending_rounds.cc:76] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 2
I20240507 08:06:03.776693 11745 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:03.790843 11745 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.776054 12060 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:03.777101 12070 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.777540 11753 raft_consensus.cc:3032] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:03.746899 11830 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.779652 10341 pending_rounds.cc:97] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240507 08:06:03.708323 12085 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.782792 12123 log.cc:1196] T 8f7d05e73be847e99048092c9570faed P 59382e6647c549d89d83c70233c0667e: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/wals/8f7d05e73be847e99048092c9570faed
I20240507 08:06:03.782620 11972 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.784897 10088 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 3, leader changed from 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:03.787701 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:03.810586 10333 raft_consensus.cc:2354] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6.
I20240507 08:06:03.811655 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.812300 10333 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:03.789014 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 126) took 1224 ms. Trace:
I20240507 08:06:03.794005 10341 raft_consensus.cc:2866] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:03.793313 12060 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:03.813714 10339 rpcz_store.cc:276] 0507 08:06:02.565009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.336428 (+771419us) service_pool.cc:225] Handling call
0507 08:06:03.363172 (+ 26744us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:03.380503 (+ 17331us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:03.380533 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:03.380584 (+ 51us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:03.380613 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.788826 (+408213us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b54003205d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.788855 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.788951 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39380864}
I20240507 08:06:03.817411 11972 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.744308 12162 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.793529 11936 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:03.796612 11745 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.797907 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1450) took 1468 ms. Trace:
I20240507 08:06:03.798466 12070 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.798869 11753 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:03.825951 10156 leader_election.cc:398] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6.
I20240507 08:06:03.826292 10156 leader_election.cc:401] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240507 08:06:03.799458 11830 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.828956 12135 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:03.793709 12137 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240507 08:06:03.798492 11744 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:03.805922 10233 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:03.832613 11744 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:03.833051 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 120208us
I20240507 08:06:03.833770 11744 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.797943 12130 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.835644 11985 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:03.811802 10088 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 2 to 4, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129), VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130), VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131), VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:03.837590 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1446) took 2035 ms. Trace:
I20240507 08:06:03.838361 11866 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:03.837802 10088 catalog_manager.cc:5484] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:03.839994 11830 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.839900 10088 catalog_manager.cc:5484] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:03.828096 10156 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:03.843020 11830 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:03.843416 11830 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:03.843735 11830 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:03.821786 10186 rpcz_store.cc:276] 0507 08:06:02.329052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.329180 (+ 128us) service_pool.cc:225] Handling call
0507 08:06:03.698971 (+1369791us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:03.706699 (+ 7728us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:03.706722 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:03.706766 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:03.706795 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.706866 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.797655 (+ 90789us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540033f4d0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:03.797754 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3097283328}
I20240507 08:06:03.845229 12179 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:03.837953 10185 rpcz_store.cc:276] 0507 08:06:01.802573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:01.803794 (+ 1221us) service_pool.cc:225] Handling call
0507 08:06:01.950202 (+146408us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:02.357162 (+406960us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.837161 (+1479999us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b540026c354. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.837417 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":934291200}
I20240507 08:06:03.831764 11936 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.867331 12051 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:03.868036 12051 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.869149 10263 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 6. Ops: [4.5-4.5]
I20240507 08:06:03.869760 12130 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:03.863121 10088 catalog_manager.cc:5484] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 4, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:03.872766 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:03.857925 11936 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:03.895812 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.901120 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:03.901928 10258 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:03.870218 12175 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:03.912490 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:03.880074 12123 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55810 (request call id 364) took 2264 ms. Trace:
I20240507 08:06:03.906380 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:03.913673 10088 catalog_manager.cc:5484] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:03.922917 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:03.928635 12053 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:03.929366 12053 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.929267 10088 catalog_manager.cc:5484] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:03.888557 11745 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:03.947448 10263 raft_consensus.cc:2379] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:03.933409 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 127) took 1368 ms. Trace:
I20240507 08:06:03.948472 10332 rpcz_store.cc:276] 0507 08:06:02.565268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.356422 (+791154us) service_pool.cc:225] Handling call
0507 08:06:03.356998 (+ 576us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:03.390293 (+ 33295us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:03.390329 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:03.395065 (+ 4736us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:03.395118 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.933177 (+538059us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b5400069d54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:03.933237 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:03.933354 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54952320}
I20240507 08:06:03.933004 12138 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:04.039098 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:04.040047 10332 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:03.873087 11753 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:03.978758 12156 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:03.925626 12175 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:03.976172 11745 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.013428 12138 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:04.081820 12052 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:03.980703 12051 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.086136 12135 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6.
I20240507 08:06:04.315121 12156 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.220405 10188 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:04.044063 10088 catalog_manager.cc:5484] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 2 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20240507 08:06:04.090472 12198 log.cc:924] Time spent T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.169s user 0.000s sys 0.000s
W20240507 08:06:04.105944 12182 log.cc:924] Time spent T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240507 08:06:04.311587 11753 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:04.223337 12137 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:04.319131 12138 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:04.340584 10088 catalog_manager.cc:5484] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 3, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:04.327582 12183 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.314510 12052 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.345996 12164 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:04.358093 12164 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.358527 12052 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.327697 12053 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:03.919752 12123 rpcz_store.cc:276] 0507 08:06:01.594485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:01.594727 (+ 242us) service_pool.cc:225] Handling call
0507 08:06:02.105695 (+510968us) ts_tablet_manager.cc:1120] Deleting tablet 8f7d05e73be847e99048092c9570faed
0507 08:06:02.105754 (+ 59us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:06:03.638191 (+1532437us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:03.775601 (+137410us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:03.858513 (+ 82912us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":148714,"tablet-delete.queue_time_us":510111,"thread_start_us":375982,"threads_started":1}
I20240507 08:06:04.338034 12070 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:04.371218 10088 catalog_manager.cc:5484] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:04.375099 12069 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:04.375633 11741 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:04.375686 12069 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.376402 11741 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:04.378473 11741 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:04.381407 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.381675 12054 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.382185 10259 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:04.382301 12054 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.383708 10155 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:04.384080 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.384440 12054 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.352011 12051 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.392441 10332 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:04.378510 12156 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.364709 11745 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:04.394321 10334 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.011s sys 0.023s
I20240507 08:06:04.394650 10156 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.342229 11830 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.393760 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:04.404433 10334 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.012s sys 0.023s
I20240507 08:06:04.409482 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.412961 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 117) took 2098 ms. Trace:
I20240507 08:06:04.413421 10187 rpcz_store.cc:276] 0507 08:06:02.314046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.314216 (+ 170us) service_pool.cc:225] Handling call
0507 08:06:02.315312 (+ 1096us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:03.642454 (+1327142us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:04.407984 (+765530us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54003b4f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:04.412876 (+ 4892us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3048837120}
I20240507 08:06:04.360616 12183 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:04.425362 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 209466us
W20240507 08:06:04.426276 10335 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s user 0.019s sys 0.063s
W20240507 08:06:04.430634 10335 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.019s sys 0.067s
I20240507 08:06:04.368088 12137 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.433606 10332 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20240507 08:06:04.404690 12070 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.425994 12069 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.434772 10332 pending_rounds.cc:76] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 2
I20240507 08:06:04.431746 11861 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:04.434190 11654 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:04.439323 11848 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:04.416615 12051 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.442487 10332 pending_rounds.cc:97] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20240507 08:06:04.446890 10332 raft_consensus.cc:2866] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:04.443126 11654 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:04.448315 11654 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:04.449937 11861 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.452155 11861 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:04.090886 12094 consensus_queue.cc:1159] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" responder_term: 6 status { last_received { term: 2 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 6. Ops: [4.5-4.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20240507 08:06:04.453073 12054 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:04.454383 12060 consensus_meta.cc:287] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.631s user 0.006s sys 0.005s
I20240507 08:06:04.435637 12164 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.478170 12175 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.478202 12044 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:04.539850 12051 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.538199 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.538585 12137 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.541299 12070 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.540108 12044 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.543577 12044 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.543483 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.543936 12188 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:04.544411 12188 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:04.544920 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.545271 12188 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.546056 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.547086 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.548064 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.550024 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.551160 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.553539 10184 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:04.554349 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1090) took 1081 ms. Trace:
I20240507 08:06:04.554708 10184 rpcz_store.cc:276] 0507 08:06:03.473261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:04.552979 (+1079718us) service_pool.cc:225] Handling call
0507 08:06:04.553326 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:04.554068 (+ 742us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:04.554290 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:04.556118 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.556932 10187 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:04.558131 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.560055 10186 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240507 08:06:04.561367 12214 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240507 08:06:04.562383 10190 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:04.502401 12060 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.567495 12060 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.568481 11654 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.569842 10266 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240507 08:06:04.572110 10192 raft_consensus.cc:1261] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:04.578438 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.453770 12054 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.585907 10339 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:04.445616 11848 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:04.584060 12211 log.cc:924] Time spent T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.142s user 0.000s sys 0.001s
I20240507 08:06:04.596271 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:04.597328 12217 log.cc:924] Time spent T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20240507 08:06:04.634526 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:04.636036 12054 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.596035 12060 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:04.636667 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 223038us
I20240507 08:06:04.638072 10156 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.639328 11754 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:04.602721 12151 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.604734 11654 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.617339 12183 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.618319 11830 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.620724 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.621767 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.521603 12164 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20240507 08:06:04.656044 10192 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:04.624614 11848 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.626081 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.657218 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:04.627625 12138 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.628432 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.657836 10267 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:04.629177 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:04.629870 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.631438 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.632164 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.633219 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.659826 10184 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:04.643375 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.648937 10338 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:04.641687 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.651623 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.637068 12060 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.626977 10188 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:04.664103 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:04.649209 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.674402 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.669875 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.676883 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.677547 10338 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:04.679088 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:04.681015 10191 raft_consensus.cc:2354] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
I20240507 08:06:04.682045 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.682091 10155 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.682765 10192 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:06:04.684108 11853 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:04.684638 11853 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:04.685083 11853 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:04.686081 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.686909 11941 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:04.687417 12183 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:04.688424 10187 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:04.687469 10304 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.689316 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.690109 10187 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:04.690255 10155 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.690887 12128 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:04.691018 11962 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:04.691277 12085 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:04.692059 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.693166 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.694249 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.695315 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:04.695827 10188 raft_consensus.cc:2362] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5. Re-sending same reply.
I20240507 08:06:04.696616 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:04.697937 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.698143 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1035) took 2094 ms. Trace:
I20240507 08:06:04.699095 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.698526 10336 rpcz_store.cc:276] 0507 08:06:02.603735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.401781 (+798046us) service_pool.cc:225] Handling call
0507 08:06:03.431534 (+ 29753us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:06:03.438796 (+ 7262us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:03.438815 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:06:03.497795 (+ 58980us) log.cc:842] Serialized 52 byte log entry
0507 08:06:03.608521 (+110726us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:03.620347 (+ 11826us) log.cc:842] Serialized 14 byte log entry
0507 08:06:03.628527 (+ 8180us) log.cc:842] Serialized 14 byte log entry
0507 08:06:03.628700 (+ 173us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:03.628763 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:03.649719 (+ 20956us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b4800180258. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:06:04.694523 (+1044804us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b18003f6b20. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd1b8 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:06:04.697920 (+ 3397us) raft_consensus.cc:1671] finished
0507 08:06:04.697944 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:04.698074 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28960,"prepare.queue_time_us":32161,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":64,"raft.queue_time_us":17522,"raft.run_cpu_time_us":2600,"raft.run_wall_time_us":40385,"spinlock_wait_cycles":43328640,"thread_start_us":110301,"threads_started":1,"wal-append.queue_time_us":141308}
I20240507 08:06:04.700649 12175 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:04.701351 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.701486 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:04.701970 10185 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:04.702126 10334 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:04.703163 11852 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:04.704608 12060 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.711038 10076 catalog_manager.cc:4830] TS 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): tablet 8f7d05e73be847e99048092c9570faed (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:04.711191 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:04.711840 10261 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:04.713276 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:04.714874 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.715971 10156 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.720027 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.721390 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.748591 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:04.765393 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 49547.8us
W20240507 08:06:04.772222 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 53735.9us
I20240507 08:06:04.772117 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.768985 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.778651 10267 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:04.784852 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.792959 10259 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:04.793473 10190 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:04.803622 10187 raft_consensus.cc:2379] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:04.773979 12094 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.805327 10188 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:04.687434 11941 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:04.808161 10259 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:04.814594 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.815485 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.815851 10334 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:04.788949 12183 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:04.793226 11745 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:04.822922 11745 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.824657 11941 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:04.824939 11745 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.825282 11744 consensus_queue.cc:1046] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:04.825542 11837 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:04.826261 11744 raft_consensus.cc:3032] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:04.826619 11744 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:04.827479 11744 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.828593 11744 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:04.830124 11992 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:04.821566 12138 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:04.821358 11959 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20240507 08:06:04.822158 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.819607 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:04.833400 10231 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:04.840062 10261 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:04.841410 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.847628 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.850411 10155 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:04.855509 10259 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240507 08:06:04.821606 11957 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:04.857352 11852 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:04.857637 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.857777 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.859041 10156 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:04.861953 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:04.863999 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.819428 11962 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.868275 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.867597 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.874398 12136 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.874938 12136 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.876441 12179 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:04.877226 12136 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.879848 10334 raft_consensus.cc:2379] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:04.880056 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:04.879408 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.945678 10184 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
W20240507 08:06:04.883478 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 262038us
I20240507 08:06:04.887436 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:04.893932 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:04.948616 10266 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:04.909945 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:04.879096 10338 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:06:04.884737 10304 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:04.952575 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 67924.8us
I20240507 08:06:04.857790 11852 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:04.960248 10336 raft_consensus.cc:2379] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:04.961941 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:04.976872 10267 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:04.980665 10335 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:04.904660 11751 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.082s
I20240507 08:06:04.900795 11957 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:04.910418 11973 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:04.870347 11941 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:04.987879 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:04.988396 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45186.4us
I20240507 08:06:04.883338 11879 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:04.982604 11846 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:04.825548 12094 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.983839 10304 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:05.058115 12132 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:05.058848 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 75105.6us
I20240507 08:06:05.058951 12132 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.060086 12054 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:05.060559 12054 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:04.884356 12044 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:05.061659 12129 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:04.840950 12183 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.064164 12183 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:04.998646 10183 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 5. Ops: []
I20240507 08:06:05.061384 12054 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.878185 11962 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:05.066360 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:05.067395 11962 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:04.843524 12188 consensus_queue.cc:1046] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.105s
I20240507 08:06:04.865729 11745 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20240507 08:06:04.985071 11852 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:04.878324 12175 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20240507 08:06:04.987350 11936 consensus_queue.cc:1046] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20240507 08:06:05.075493 11936 raft_consensus.cc:3032] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:05.075837 11936 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:05.076654 11936 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:04.977150 12070 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20240507 08:06:05.054534 10337 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:05.083163 11853 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.082285 11936 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.047267 11846 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:05.046962 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:05.046774 11973 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.058209 12094 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.061353 10304 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:05.087949 11852 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.088716 12159 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:05.000130 11957 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:05.089473 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 49346.2us
I20240507 08:06:05.089506 11941 raft_consensus.cc:686] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:05.090579 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:05.091706 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:05.093439 10939 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240507 08:06:05.139426 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:05.141609 11853 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.093941 12044 raft_consensus.cc:686] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:05.143415 11852 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.143133 12044 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.096975 10158 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:05.146171 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 49496.9us
I20240507 08:06:05.147300 10158 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.148016 12052 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:05.148406 10158 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.098289 10266 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:06:05.080195 11869 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:05.150956 12183 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240507 08:06:05.113427 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 63989.2us
I20240507 08:06:05.151399 10262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1107) took 1279 ms. Trace:
I20240507 08:06:05.151804 10262 rpcz_store.cc:276] 0507 08:06:03.871691 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.871862 (+ 171us) service_pool.cc:225] Handling call
0507 08:06:03.890634 (+ 18772us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:03.907040 (+ 16406us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:03.907085 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:03.921660 (+ 14575us) log.cc:842] Serialized 26 byte log entry
0507 08:06:04.542382 (+620722us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b1800373060. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:06:04.801053 (+258671us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:04.803210 (+ 2157us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:04.810583 (+ 7373us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:05.151146 (+340563us) raft_consensus.cc:1671] finished
0507 08:06:05.151167 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:05.151346 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13560,"raft.queue_time_us":12501,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":7292,"spinlock_wait_cycles":4563200,"thread_start_us":258491,"threads_started":1,"wal-append.queue_time_us":1084172}
I20240507 08:06:05.048599 12049 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:05.154341 12049 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.061426 12132 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.156802 12049 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.129144 10192 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:05.131939 11788 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:05.130781 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 69502us
I20240507 08:06:05.176802 10158 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.181958 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.164836 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.170405 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:05.167045 10264 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:05.183705 10264 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:05.122262 11846 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.112897 12175 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:05.165736 11837 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:05.189188 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.190218 10266 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
W20240507 08:06:05.190546 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 317776us
I20240507 08:06:05.190778 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.191535 10261 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:05.186683 10264 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.192888 10264 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:05.195952 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 331107us
I20240507 08:06:05.196723 10232 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.198010 10232 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.199857 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.202309 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:05.186347 11788 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.194260 11837 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:05.220008 11837 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:05.220070 12175 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.222334 12175 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.223237 10940 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 6. Ops: []
I20240507 08:06:05.224264 11952 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20240507 08:06:05.172518 12159 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.207296 12060 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:05.179914 12052 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:05.226426 11853 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240507 08:06:05.239787 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 63148.5us
I20240507 08:06:05.190814 12151 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:05.241761 11973 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.243562 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:05.266810 10334 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:05.244819 10333 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240507 08:06:05.246220 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 44003.9us
I20240507 08:06:05.246174 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.268460 10336 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:05.200974 11601 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:05.217635 12231 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.095s user 0.000s sys 0.001s
I20240507 08:06:05.201198 12128 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:05.208930 11985 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:05.234089 11941 consensus_queue.cc:237] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.216288 10267 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
W20240507 08:06:05.248378 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 52900.8us
I20240507 08:06:05.291237 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:05.233919 11750 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.009s sys 0.005s
I20240507 08:06:05.246819 12136 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:05.257057 10264 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240507 08:06:05.285624 10340 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:05.303260 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:05.286275 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 86960.6us
I20240507 08:06:05.303318 10264 pending_rounds.cc:76] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 3
I20240507 08:06:05.304193 10335 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:05.304383 10264 pending_rounds.cc:97] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240507 08:06:05.304529 10232 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.304806 10264 raft_consensus.cc:2866] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240507 08:06:05.309196 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 533504us
I20240507 08:06:05.304922 10341 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:05.312343 10155 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:05.313491 10340 raft_consensus.cc:1261] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20240507 08:06:05.314946 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.267851 12060 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:05.320153 10232 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:05.322295 10341 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 1174) took 1473 ms (client timeout 1548 ms). Trace:
W20240507 08:06:05.327986 10341 rpcz_store.cc:269] 0507 08:06:03.849241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:03.859374 (+ 10133us) service_pool.cc:225] Handling call
0507 08:06:05.322107 (+1462733us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54003c4954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:05.322232 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3258952064}
I20240507 08:06:05.328672 10188 raft_consensus.cc:2379] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:05.330762 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:05.331997 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 43612.2us
W20240507 08:06:05.302918 11750 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.696s user 0.010s sys 0.005s
I20240507 08:06:05.339533 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.341107 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.342547 10186 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:05.340499 10192 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:05.346442 10191 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:05.358258 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:05.308233 12136 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:05.367283 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.370100 10334 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:05.319352 12069 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:05.273499 12179 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20240507 08:06:05.375370 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.345638 11848 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:05.331535 11744 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:05.334686 12060 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:05.273371 12151 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.354324 11753 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:05.382541 11848 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:05.383641 11920 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:05.384467 12132 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240507 08:06:05.384946 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52108.8us
I20240507 08:06:05.388129 11754 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240507 08:06:05.388237 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 76005.7us
I20240507 08:06:05.374603 12136 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:05.405534 10155 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:05.383646 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 63600.7us
I20240507 08:06:05.406571 10232 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.304080 11601 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.286124 12128 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.335899 11788 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.404749 11754 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:05.415159 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.381191 12069 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:05.416435 10232 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.428320 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.431344 10232 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.412516 11973 raft_consensus.cc:3032] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:05.396822 11959 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20240507 08:06:05.436548 10232 leader_election.cc:398] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
I20240507 08:06:05.414479 11962 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:05.417145 11754 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.437255 10232 leader_election.cc:401] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:05.417423 12069 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.442504 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.444555 10191 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240507 08:06:05.431650 12156 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:05.428433 12132 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:05.405277 11753 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.424445 12245 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:05.429548 12070 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:05.429898 11914 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:05.437096 12175 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:05.436197 11973 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:05.406888 11920 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.472362 12138 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:05.473942 12138 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:05.474447 12138 raft_consensus.cc:686] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:05.475353 12138 consensus_queue.cc:237] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.477799 12137 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:05.478192 11914 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.480475 11914 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.465770 11852 consensus_queue.cc:1046] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240507 08:06:05.480178 10156 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.481662 11601 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.481786 12164 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:05.482232 12164 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:05.482511 11852 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:05.482612 12164 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.483659 12128 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.485129 10156 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:05.482913 11852 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:05.485993 12160 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:05.458039 12179 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5.
W20240507 08:06:05.488739 12228 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.018s sys 0.036s
W20240507 08:06:05.489071 12228 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.873s user 0.018s sys 0.037s
I20240507 08:06:05.489236 12137 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:05.461705 12245 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:05.489846 12137 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:05.490411 12245 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.482730 11848 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.482749 11853 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:05.492228 11853 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:05.500000 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.485083 11972 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:05.524344 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.527704 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.537791 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.539566 10191 raft_consensus.cc:2379] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:05.543871 10263 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20240507 08:06:05.486369 11852 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.551645 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.486400 12160 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:05.561718 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.506522 11601 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:05.571666 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.524605 12094 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:05.489094 11837 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.591615 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.598423 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.590559 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.608808 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.612469 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.613163 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:05.615374 11920 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.618286 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.615010 11957 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.621609 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:05.567209 12160 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:05.624984 12050 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:05.625742 11914 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:05.627038 12159 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.628674 10263 raft_consensus.cc:1226] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:05.637821 10259 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240507 08:06:05.620756 12250 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.096s user 0.000s sys 0.001s
I20240507 08:06:05.633796 10232 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.645643 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.647182 10232 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:05.648166 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:05.650394 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 96987.6us
I20240507 08:06:05.641155 11853 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:05.661494 10306 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.645712 11745 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:05.649736 12175 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240507 08:06:05.681082 12242 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20240507 08:06:05.679190 11920 consensus_queue.cc:1046] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20240507 08:06:05.695437 12060 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.598064 11973 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.642078 12050 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.697954 10259 raft_consensus.cc:1226] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:05.703285 10306 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.697531 12136 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.704931 12137 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:05.704975 11846 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.786499 12136 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.787705 11846 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.710139 11973 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:05.792510 11852 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:05.755378 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:05.806865 10258 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:05.724869 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:05.812990 11601 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:05.813931 11601 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.815572 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:05.737512 12238 log.cc:924] Time spent T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240507 08:06:05.702086 11829 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:05.825961 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.826887 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.702545 11920 raft_consensus.cc:3032] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:05.832973 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.739311 12151 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.740789 11837 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:05.703953 12245 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.836477 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.731843 11753 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.764420 11914 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:05.735527 12050 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:05.827960 11985 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:05.701839 11992 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.134s
I20240507 08:06:05.743418 12069 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.831779 11920 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:05.740041 12164 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:05.860649 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 157473us
I20240507 08:06:05.872493 10306 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.763499 11754 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:05.877411 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52437.2us
I20240507 08:06:05.879104 10262 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:05.850193 12257 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:05.876055 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:05.914412 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:05.915573 10337 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:05.916935 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:05.918224 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:05.918287 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 45906.7us
I20240507 08:06:05.919157 12129 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:05.919103 10306 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.921792 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:05.922010 10306 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.923578 11936 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:05.924088 11936 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:05.924908 11936 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:05.925881 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:05.926415 10266 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:05.928120 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:05.928864 11788 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:05.929224 11788 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:05.929551 11788 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:05.931222 10306 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.932505 10306 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.934880 10258 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:05.942824 10190 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:05.942831 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:05.864930 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:05.935154 12230 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20240507 08:06:06.010952 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1113) took 1662 ms. Trace:
I20240507 08:06:06.015347 12069 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.015386 11957 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:06.016599 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50543.3us
I20240507 08:06:05.881158 12132 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:05.921814 12159 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.019807 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:06.025904 10267 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:06.022207 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 79490.1us
W20240507 08:06:06.017314 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 86358.8us
I20240507 08:06:06.017534 10306 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.017518 10260 rpcz_store.cc:276] 0507 08:06:04.348490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:04.348614 (+ 124us) service_pool.cc:225] Handling call
0507 08:06:04.367113 (+ 18499us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:04.644098 (+276985us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:04.644128 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:04.663457 (+ 19329us) log.cc:842] Serialized 26 byte log entry
0507 08:06:05.156707 (+493250us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:05.156934 (+ 227us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:05.157889 (+ 955us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:06.010719 (+852830us) raft_consensus.cc:1671] finished
0507 08:06:06.010741 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:06.010899 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30227,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":18308,"thread_start_us":683206,"threads_started":2,"wal-append.queue_time_us":898850}
I20240507 08:06:05.954931 11846 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.054353 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:06.056741 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:06.058179 10306 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:05.956243 12160 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.094149 10306 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.024574 11920 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.103897 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:06.107018 10267 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:06.052255 12255 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:06.098479 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:06.156391 10332 raft_consensus.cc:2354] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:06.094189 11837 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:06.122768 12060 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.129928 11920 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:06.097755 12257 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:06.095510 10266 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:06.093006 12159 consensus_queue.cc:1046] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240507 08:06:06.097959 12245 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.102185 11846 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.106038 12164 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:06.141549 10334 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.007s sys 0.021s
W20240507 08:06:06.193192 10334 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s user 0.007s sys 0.021s
I20240507 08:06:06.156977 10304 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:06.189126 10907 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.008s sys 0.027s
W20240507 08:06:06.201303 10907 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.008s sys 0.027s
I20240507 08:06:06.178545 11985 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:06.213551 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 56701.8us
I20240507 08:06:06.215483 10304 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:06.221292 10231 leader_election.cc:398] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:06.221707 10231 leader_election.cc:401] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240507 08:06:06.240260 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.206367 12136 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:06.252334 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.261929 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.264293 10189 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
W20240507 08:06:06.268332 12264 log.cc:924] Time spent T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.124s user 0.000s sys 0.000s
I20240507 08:06:06.268095 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.269138 10183 raft_consensus.cc:2354] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:06.249027 11985 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.274948 10232 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.275544 11985 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.276094 11985 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:06.277123 12160 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.278825 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:06.282298 12265 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20240507 08:06:06.283504 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1379) took 1102 ms. Trace:
I20240507 08:06:06.283826 10265 rpcz_store.cc:276] 0507 08:06:05.188303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:05.206156 (+ 17853us) service_pool.cc:225] Handling call
0507 08:06:05.269521 (+ 63365us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:06:05.340973 (+ 71452us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:05.341017 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:06:05.374893 (+ 33876us) log.cc:842] Serialized 52 byte log entry
0507 08:06:05.647550 (+272657us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:05.647591 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:05.647653 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:06.283315 (+635662us) raft_consensus.cc:1671] finished
0507 08:06:06.283335 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:06.283454 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41203,"raft.queue_time_us":36039,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":17279,"thread_start_us":240815,"threads_started":1,"wal-append.queue_time_us":476704}
I20240507 08:06:06.289371 10232 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.290829 12132 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:06:06.278956 12256 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:06.296521 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.299127 10232 leader_election.cc:398] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:06.299465 10232 leader_election.cc:401] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240507 08:06:06.301076 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1198) took 1359 ms. Trace:
I20240507 08:06:06.302222 10158 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.308394 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.252197 12245 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:06.302974 10338 rpcz_store.cc:276] 0507 08:06:04.942242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:04.956503 (+ 14261us) service_pool.cc:225] Handling call
0507 08:06:04.988039 (+ 31536us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:04.988450 (+ 411us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:04.988490 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:04.989001 (+ 511us) log.cc:842] Serialized 26 byte log entry
0507 08:06:05.195850 (+206849us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b18000b4280. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:06:05.332768 (+136918us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:05.381400 (+ 48632us) log.cc:842] Serialized 14 byte log entry
0507 08:06:06.044776 (+663376us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b18000b4280. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8def9cc5 00007fcf90e1f620 00007fcf90e317c6 00007fcf90e42327 00007fcf90e422a8
0507 08:06:06.300685 (+255909us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:06.300776 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:06.300837 (+ 61us) raft_consensus.cc:1671] finished
0507 08:06:06.300861 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:06.301006 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6119,"prepare.queue_time_us":46360,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":34326,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":62747,"spinlock_wait_cycles":84978432,"thread_start_us":392496,"threads_started":2,"wal-append.queue_time_us":431057}
W20240507 08:06:06.334905 10155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.008s sys 0.024s
W20240507 08:06:06.335564 10155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.008s sys 0.024s
I20240507 08:06:06.297542 12159 raft_consensus.cc:3032] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:06.297116 12136 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
W20240507 08:06:06.335996 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 684958us
W20240507 08:06:06.347239 10184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.023s sys 0.022s
W20240507 08:06:06.350183 10184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s user 0.023s sys 0.022s
I20240507 08:06:06.313473 12175 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:06:06.362388 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 60313.1us
I20240507 08:06:06.367326 10158 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.324586 12132 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:06.370980 10155 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.329885 12256 raft_consensus.cc:686] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:06.348754 12159 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:06.311123 11788 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:06.398653 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 55616.8us
I20240507 08:06:06.364719 12175 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.365006 12136 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:06.402310 11941 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240507 08:06:06.407701 12274 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240507 08:06:06.407874 11879 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:06.408481 11925 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.408946 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 142) took 1120 ms. Trace:
I20240507 08:06:06.409107 11925 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.410518 10332 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20240507 08:06:06.411878 11925 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.412588 11985 consensus_queue.cc:1046] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:06.413707 11985 raft_consensus.cc:3032] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:06.414060 11985 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:06.414856 11985 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.416110 11985 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:06.409349 10333 rpcz_store.cc:276] 0507 08:06:05.288118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:05.290501 (+ 2383us) service_pool.cc:225] Handling call
0507 08:06:05.325977 (+ 35476us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:05.504196 (+178219us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:05.504265 (+ 69us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:05.543399 (+ 39134us) log.cc:842] Serialized 26 byte log entry
0507 08:06:06.084981 (+541582us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:06.085038 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:06.085113 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:06.408760 (+323647us) raft_consensus.cc:1671] finished
0507 08:06:06.408779 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:06.408897 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3349,"raft.queue_time_us":30638,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":4105,"thread_start_us":591998,"threads_started":2,"wal-append.queue_time_us":744999}
I20240507 08:06:06.419246 12159 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.419888 11972 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:06.420423 12159 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:06.422163 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.423086 10155 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.423867 11788 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:06.424916 12094 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:06.425202 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:06.429330 12269 log.cc:924] Time spent T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
I20240507 08:06:06.440434 10338 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:06.391295 12183 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:06.435797 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 220468us
I20240507 08:06:06.402081 12044 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.409386 11879 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:06.432186 12175 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:06.405947 12267 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.455411 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:06.462345 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 215281us
I20240507 08:06:06.469630 10192 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:06.452898 12245 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:06.456537 12060 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:06.448904 12284 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:06.478331 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:06.461019 11654 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.576548 10338 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:06.452853 12183 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:06.585523 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 112078us
W20240507 08:06:06.599350 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 120573us
I20240507 08:06:06.600813 10333 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 6. Current term is 7. Ops: []
W20240507 08:06:06.606307 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 138056us
I20240507 08:06:06.464613 12044 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.608587 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:06.611058 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:06.588083 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:06.628420 12257 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:06:06.593396 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 126281us
I20240507 08:06:06.482303 12245 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.484424 12060 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.650454 10264 raft_consensus.cc:1226] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
W20240507 08:06:06.651378 12287 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20240507 08:06:06.654875 12044 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:06.655265 10340 consensus_meta.cc:287] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.598s user 0.006s sys 0.002s
I20240507 08:06:06.655642 10340 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
W20240507 08:06:06.656826 12285 log.cc:924] Time spent T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240507 08:06:06.475446 12267 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.658682 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1123) took 1046 ms. Trace:
I20240507 08:06:06.658915 11829 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:06.659031 10184 rpcz_store.cc:276] 0507 08:06:05.612432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:05.612726 (+ 294us) service_pool.cc:225] Handling call
0507 08:06:05.613021 (+ 295us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:05.808500 (+195479us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:05.808527 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:05.808961 (+ 434us) log.cc:842] Serialized 26 byte log entry
0507 08:06:06.350695 (+541734us) raft_consensus.cc:1631] Marking committed up to 1
0507 08:06:06.350747 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:06.350834 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:06.658488 (+307654us) raft_consensus.cc:1671] finished
0507 08:06:06.658507 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:06.658632 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":254942,"raft.queue_time_us":195411,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":2797,"thread_start_us":525031,"threads_started":1,"wal-append.queue_time_us":644161}
I20240507 08:06:06.664978 12183 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:06.593128 11654 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.667022 11654 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.481247 12256 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.637723 11750 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:06.667857 11750 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.670258 11750 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.671205 12188 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:06.672462 12162 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:06.673789 12271 raft_consensus.cc:3032] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:06.674203 12271 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:06.675061 11920 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:06.675608 11920 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.675287 12271 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:06.677044 12271 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:06.681298 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:06.683516 10138 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 59382e6647c549d89d83c70233c0667e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:06.685338 12060 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.688221 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:06.689045 10340 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:06.689837 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:06.691391 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:06.693172 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 228097us
I20240507 08:06:06.693843 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:06.694988 10306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.003s sys 0.024s
W20240507 08:06:06.706940 10306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.004s sys 0.024s
W20240507 08:06:06.707461 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 613545us
I20240507 08:06:06.708185 10306 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.708518 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.709561 12085 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:06.711413 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:06.713078 10306 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.716444 10267 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:06.677476 11920 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:06.718534 10306 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.720494 10306 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.691308 11914 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:06.607369 12136 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.726269 10306 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:06.725425 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 53300.9us
I20240507 08:06:06.732139 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:06.733340 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:06.723263 12085 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:06.746735 10264 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:06.747740 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:06.724550 12183 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:06.760108 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.760915 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.761081 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.761476 10188 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:06.764478 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.765146 10183 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:06.769325 10188 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:06.775444 10190 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:06.760670 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.778738 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:06.779552 10266 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 6: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:06.779490 10156 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:06.782665 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:06.743665 12289 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:06.783485 10187 raft_consensus.cc:2354] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5.
I20240507 08:06:06.784523 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.742705 12132 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
W20240507 08:06:06.788585 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 78765.8us
I20240507 08:06:06.799134 10183 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:06.799654 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.805701 10183 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:06.817087 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.799659 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.747839 12151 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:06.753644 11957 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.790829 12289 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:06.732264 11914 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.823014 10333 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:06.826887 10306 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.827181 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:06.861449 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.820832 10185 raft_consensus.cc:2362] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6. Re-sending same reply.
I20240507 08:06:06.866343 10266 raft_consensus.cc:2379] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:06.866874 10306 leader_election.cc:398] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5.
I20240507 08:06:06.867736 10306 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:06.871450 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 87508.9us
I20240507 08:06:06.874840 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:06.875437 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:06.876103 10266 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:06.783985 12256 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20240507 08:06:06.821599 11654 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.821228 12160 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:06.863587 12289 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:06.817836 11753 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.885389 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:06.828186 12085 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:06.862521 12267 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.894129 11846 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.896746 12292 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240507 08:06:06.892122 10266 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20240507 08:06:06.911322 10266 pending_rounds.cc:76] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 1
W20240507 08:06:06.888983 12279 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240507 08:06:06.899909 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:06.911584 10266 pending_rounds.cc:97] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20240507 08:06:06.931051 10266 raft_consensus.cc:2866] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:06.889482 10267 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:06.910946 11936 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:06.911324 12303 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:06.920897 12025 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:06.949311 12303 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.949443 12025 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:06.951587 12303 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:06.923357 12299 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
W20240507 08:06:06.924212 12297 log.cc:924] Time spent T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20240507 08:06:06.924510 12183 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.928007 11914 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:06.957892 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 68020.5us
I20240507 08:06:06.936318 12245 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.942234 11920 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:06.895601 11654 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.860311 12151 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:06.928413 12085 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.946957 12175 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.947746 10185 raft_consensus.cc:2396] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240507 08:06:06.911608 11753 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:06.952278 12025 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:06.982872 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 182214us
I20240507 08:06:06.972250 10265 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:06.974647 12151 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:06.976861 12085 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:06.978514 11654 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:06.891708 12160 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:07.033660 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.037591 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.035537 10231 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:06.984354 11754 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.044998 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.048971 10264 raft_consensus.cc:2354] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:07.034922 12160 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:07.028914 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 46930.2us
I20240507 08:06:07.032440 10266 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:07.052402 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.063452 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:07.060901 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.074036 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.008955 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.084357 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.042798 12245 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.048063 11744 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:07.095717 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.097570 10262 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
W20240507 08:06:07.110927 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 223626us
I20240507 08:06:07.111045 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.111702 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:07.118156 10231 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:07.093042 12315 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.134192 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.143709 12315 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.145757 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.123353 12151 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:07.146826 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.120429 12044 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.147600 12044 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.147853 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.147990 12289 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.146652 12151 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.152061 12151 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.153496 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.149403 12044 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.145936 12315 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.154330 10263 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:07.154506 12175 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.156811 11753 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.160663 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:07.162005 10231 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:07.162694 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:07.162849 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 50685.5us
I20240507 08:06:07.127404 10183 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:07.169431 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.166370 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.177878 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.178323 11750 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.179004 12070 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:07.178759 11750 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.180651 11750 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.181449 12315 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:07.185396 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.187654 10155 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.189505 12282 consensus_queue.cc:1046] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.187713 11750 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:07.192157 12094 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:07.192750 12094 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.194052 11957 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.194561 11957 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.160564 11744 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.196449 11957 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.196221 11744 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.197409 10341 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:07.197997 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.198653 11744 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.199782 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.204922 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.207567 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.208329 10184 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:07.175021 10259 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:07.122954 12085 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:07.095199 12245 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.214587 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:07.211493 10232 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.219249 10192 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.213299 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.201378 12286 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.169523 12069 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.223040 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:07.206687 12265 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20240507 08:06:07.171809 12129 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:07.203033 12288 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:07.230942 10908 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.231752 12056 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:07.232337 12289 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:07.234592 10262 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:07.212821 12282 raft_consensus.cc:3032] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:07.238302 12282 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:07.239316 12282 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.229722 12069 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.248113 12069 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.261054 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.226598 11957 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:07.269152 10192 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
W20240507 08:06:07.270687 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 68614.8us
W20240507 08:06:07.255548 12255 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.049s user 0.017s sys 0.051s
W20240507 08:06:07.296533 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 107853us
W20240507 08:06:07.346271 12255 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.154s user 0.018s sys 0.051s
I20240507 08:06:07.227530 12052 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20240507 08:06:07.350196 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.352885 10155 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.268071 11957 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:07.251729 12129 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:07.228147 12286 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.268419 12160 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.359426 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:07.359925 12245 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.360913 11830 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:07.356055 12070 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:07.361514 11866 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:07.363375 12129 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:07.353430 12151 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:07.364449 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.391819 12286 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:07.396721 12151 raft_consensus.cc:686] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:07.398015 12151 consensus_queue.cc:237] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.357357 11957 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.375365 10155 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:07.350621 12255 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:07.386276 12160 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.416394 10155 leader_election.cc:398] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
W20240507 08:06:07.416471 12257 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.666s user 0.010s sys 0.000s
I20240507 08:06:07.416723 10155 leader_election.cc:401] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:07.417862 12160 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:07.418247 12160 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:07.418582 12160 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:07.422503 11866 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:07.423956 11866 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:07.410681 10233 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:07.425984 12303 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:07.427500 11972 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:07.427980 10155 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.435853 11830 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:07.435271 12255 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.436921 11866 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:07.442708 11866 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.445137 11866 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.436220 10155 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.446895 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.448185 10155 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.447474 11866 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:07.449513 11866 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:07.449970 11866 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.450313 12094 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:07.451273 12094 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:07.452241 12094 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.455006 11957 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.455564 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:07.456825 10336 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:07.450522 10155 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:07.461010 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.467692 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.468695 10267 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 6: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:07.472898 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:07.474818 10155 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.477394 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.474823 11866 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:07.474735 11972 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:07.496755 11972 raft_consensus.cc:686] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:07.493676 11914 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:07.497634 11972 consensus_queue.cc:237] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.498015 11914 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.498996 12281 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:07.499912 11914 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.502022 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.502719 10191 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:07.504284 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.505379 12312 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:07.477543 12267 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:07.435434 12160 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.505818 12312 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.538244 11957 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.540735 12160 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.543951 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:07.532395 12312 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:07.556046 12295 log.cc:924] Time spent T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.726s user 0.011s sys 0.003s
I20240507 08:06:07.527204 12303 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:07.559315 12303 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:07.561226 12303 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:07.564611 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 89882us
I20240507 08:06:07.527017 12267 raft_consensus.cc:686] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:07.568677 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.569401 10267 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:07.584553 10267 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:07.588891 10155 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.597180 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:07.598768 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.588197 11866 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.613993 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:07.616353 10233 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:07.617676 12303 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:07.614109 11866 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:07.591051 11837 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:07.624294 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.624768 10267 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:07.623279 12267 consensus_queue.cc:237] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.624212 11837 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.643344 11936 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:07.643824 11936 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:07.645781 12330 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:07.646348 12330 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:07.648423 12330 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.649761 12282 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.656157 10267 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:07.659241 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.660962 12052 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:07.663120 11837 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.666515 11829 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:07.667035 11829 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:07.668915 11837 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:07.672097 12312 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.675109 12052 raft_consensus.cc:686] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:07.683769 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 161) took 1922 ms. Trace:
I20240507 08:06:07.684085 12311 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:07.682863 12052 consensus_queue.cc:237] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.687618 12311 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:07.696794 11829 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.698645 11829 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:07.710698 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:07.711313 10186 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:07.724701 12262 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.598s user 0.016s sys 0.017s
I20240507 08:06:07.738718 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:07.742245 12262 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.616s user 0.016s sys 0.017s
I20240507 08:06:07.732308 12312 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:07.822506 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 72230.9us
W20240507 08:06:07.823364 12350 log.cc:924] Time spent T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20240507 08:06:07.826004 10186 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
W20240507 08:06:07.832299 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 81643.6us
I20240507 08:06:07.837278 10232 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:07.843835 12311 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:07.844244 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 679943us
I20240507 08:06:07.844909 10304 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.845386 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:07.843572 12256 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.862529 10185 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20240507 08:06:07.864099 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 592933us
I20240507 08:06:07.864792 10306 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:07.842337 12132 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:07.874091 12054 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:07.866978 12289 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:07.890638 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 44735us
I20240507 08:06:07.886679 12132 raft_consensus.cc:686] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:07.905062 10186 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:07.907119 10186 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:07.882246 12070 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240507 08:06:07.908254 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:07.908886 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:07.911355 10189 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:07.909670 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:07.914410 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.921322 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.923789 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.924065 12129 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.918262 12355 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.007s
I20240507 08:06:07.925431 12159 consensus_queue.cc:1046] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.924465 10258 raft_consensus.cc:2354] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5.
I20240507 08:06:07.926595 12159 raft_consensus.cc:3032] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:07.927003 12159 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:07.927847 12159 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.928611 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.928978 12159 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:07.929708 12355 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.930332 12355 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.932662 12355 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:07.936236 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:07.937165 10258 raft_consensus.cc:2379] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:07.940155 10258 raft_consensus.cc:1226] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:07.943521 10266 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:07.944070 10258 raft_consensus.cc:1261] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:07.947571 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.952250 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:07.944370 11846 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.959553 11846 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.964251 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.950726 12286 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:07.969591 12286 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.970608 12340 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:07.971555 12286 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:07.973764 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 534af8458dc14ec49ca7308cf02b685c, TS bcf08c8276ba4a9ba646ad0f7090fcd8 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:07.974103 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 534af8458dc14ec49ca7308cf02b685c, TS bcf08c8276ba4a9ba646ad0f7090fcd8 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:07.969926 12352 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20240507 08:06:07.957540 12132 consensus_queue.cc:237] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.684537 10261 rpcz_store.cc:276] 0507 08:06:05.761511 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:05.761663 (+ 152us) service_pool.cc:225] Handling call
0507 08:06:05.762034 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:05.762283 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 6
0507 08:06:07.657773 (+1895490us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b180036b140. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf90ab675c 00007fcf8dedf54c 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8def9cc5 00007fcf947fdf17 00007fcf90e31a57 00007fcf90e4115f
0507 08:06:07.657905 (+ 132us) log.cc:842] Serialized 14 byte log entry
0507 08:06:07.683271 (+ 25366us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:07.683335 (+ 64us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:07.683401 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:07.683556 (+ 155us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:07.683684 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39385,"prepare.queue_time_us":458208,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":82,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":1543,"spinlock_wait_cycles":2550016,"thread_start_us":505610,"threads_started":2}
I20240507 08:06:07.981195 11846 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:07.983361 11936 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:07.985911 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:07.986318 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:07.987385 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:07.985340 11936 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:08.004995 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.005615 10336 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.005905 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.008055 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.008886 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.010218 11829 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:08.013738 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:08.003177 10334 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:08.017943 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.018529 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.018666 10333 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:08.020190 11829 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.022601 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.024121 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.037118 12289 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:08.026798 11899 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:08.038872 11829 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:08.045972 10341 raft_consensus.cc:1226] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:08.047619 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.048137 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.041787 12085 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:08.045114 10338 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:08.060658 10336 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:08.072193 10332 raft_consensus.cc:1226] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:08.083880 10332 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:08.087181 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45982.1us
I20240507 08:06:08.087898 10156 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.049058 11899 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:08.104013 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:08.104296 12353 log.cc:924] Time spent T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20240507 08:06:08.104687 10336 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:08.105463 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.071401 12085 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.105963 10336 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:08.122823 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.110514 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.106477 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.106091 10333 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:08.113042 10266 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:08.110145 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.111709 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:08.124562 10231 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.126065 10267 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:08.127383 10260 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:08.131260 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:08.137327 10231 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.138921 10231 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.097501 12288 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:08.140247 10336 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:08.141319 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.142848 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.144994 10304 leader_election.cc:398] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5.
I20240507 08:06:08.145427 10304 leader_election.cc:401] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:08.136155 11745 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:08.135982 11829 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:08.149442 11829 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.151577 11829 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:08.151983 11899 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:08.153415 12288 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.142889 12132 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:08.155539 12288 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:08.144186 12331 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:08.156407 12085 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:08.157498 12132 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:08.158206 12132 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.160508 12132 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.146363 10304 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.162003 12129 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:08.163049 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.164050 12129 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:08.165957 12129 consensus_queue.cc:1046] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:08.167081 12085 raft_consensus.cc:3032] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:08.167425 12085 raft_consensus.cc:729] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:08.168329 12085 consensus_queue.cc:260] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:08.170892 12357 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.206s user 0.000s sys 0.004s
W20240507 08:06:08.174214 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 49815.5us
I20240507 08:06:08.174754 12085 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:08.175354 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.142629 12303 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:08.180660 10266 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:08.180979 10260 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:08.132925 11866 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:08.181960 11866 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:08.182329 11866 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:08.182518 10304 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.183045 11941 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20240507 08:06:08.184254 11973 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:08.144335 12151 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:08.186153 12151 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:08.180971 10156 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.186620 12151 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.187024 11853 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:08.143996 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.187423 10156 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.187942 12303 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:08.188519 11750 raft_consensus.cc:2741] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20240507 08:06:08.189317 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45396.4us
I20240507 08:06:08.190886 12303 consensus_queue.cc:1046] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:08.192049 12303 raft_consensus.cc:3032] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:08.192385 12303 raft_consensus.cc:729] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:08.193475 12303 consensus_queue.cc:260] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.195791 12303 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:08.157716 12359 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240507 08:06:08.148970 12319 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5.
I20240507 08:06:08.207913 10304 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.145054 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.214290 10264 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:08.216477 10333 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:08.217989 10341 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.220191 10192 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:08.227618 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.203497 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.229790 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.231675 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.233464 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:08.235002 10334 raft_consensus.cc:2379] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:08.238262 10338 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:08.238682 10338 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:08.239557 10338 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.244473 10338 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.247421 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.243600 12319 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:08.230587 12129 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:08.252719 10304 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.234197 10940 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:08.230474 12069 consensus_queue.cc:1046] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20240507 08:06:08.229486 11853 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20240507 08:06:08.271751 10158 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.264397 10338 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20240507 08:06:08.275022 10338 pending_rounds.cc:76] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 3
I20240507 08:06:08.275310 10338 pending_rounds.cc:97] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20240507 08:06:08.275589 10338 raft_consensus.cc:2866] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:08.264689 10304 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.277465 11899 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240507 08:06:08.248909 11788 consensus_queue.cc:1046] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240507 08:06:08.266562 12319 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:08.272404 12312 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:08.277881 12284 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:08.270279 12288 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20240507 08:06:08.312512 11866 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.318310 12354 consensus_queue.cc:1046] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:08.271345 11829 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20240507 08:06:08.306917 10304 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.292188 12286 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:08.287775 12340 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20240507 08:06:08.304087 12284 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:08.337721 12365 log.cc:924] Time spent T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
W20240507 08:06:08.362356 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 55586.2us
I20240507 08:06:08.371371 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.372943 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.374487 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.299959 12312 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.293196 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "324b71380c6c47bbb335661d299baea0"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:08.381963 10185 raft_consensus.cc:595] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.388967 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.391647 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.396855 10338 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:08.399780 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.366505 12354 consensus_queue.cc:1225] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20240507 08:06:08.406080 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 19878aa675724839ac9bb59333dee2d9, TS bcf08c8276ba4a9ba646ad0f7090fcd8 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:08.406422 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 19878aa675724839ac9bb59333dee2d9, TS bcf08c8276ba4a9ba646ad0f7090fcd8 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.395761 10304 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.367852 12183 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:08.379115 12284 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:08.389953 12245 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:08.408092 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.412899 10304 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.416707 12085 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:08.398160 11745 consensus_queue.cc:1046] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240507 08:06:08.412521 11920 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:08.437914 10341 raft_consensus.cc:1261] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:08.421350 12136 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240507 08:06:08.434594 12368 log.cc:924] Time spent T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20240507 08:06:08.466667 11866 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.440371 11788 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:08.466820 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:08.487390 10265 raft_consensus.cc:2379] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 3: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:08.496819 10333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:08.497568 10333 raft_consensus.cc:595] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.498021 10333 raft_consensus.cc:603] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:08.500180 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.468856 12050 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:08.504936 12050 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.506966 12050 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:08.508901 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 117620us
I20240507 08:06:08.511526 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.511873 10188 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:08.512118 10185 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:08.519451 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:08.520057 10266 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:08.521205 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:08.522248 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.522845 10338 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:08.523550 10341 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:08.462064 12373 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240507 08:06:08.529088 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 137583us
I20240507 08:06:08.532791 10232 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:08.534173 12379 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:08.534976 12372 log.cc:924] Time spent T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20240507 08:06:08.534698 12379 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.536218 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:08.536785 10185 raft_consensus.cc:595] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.463184 12340 consensus_queue.cc:1035] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20240507 08:06:08.537701 12312 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.539953 10156 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:08.548904 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b909813a654d41cf88fb9fee2f8006bc, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:08.549243 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b909813a654d41cf88fb9fee2f8006bc, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:08.554886 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3490f6d2f719474bb4f72a5647a9a726, TS 88a26dac37fa4243b8c2b3d47f403488 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:08.555238 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3490f6d2f719474bb4f72a5647a9a726, TS 88a26dac37fa4243b8c2b3d47f403488 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.577994 10340 raft_consensus.cc:3032] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:08.578379 10340 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:08.580265 10341 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:08.580976 10341 raft_consensus.cc:595] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.581331 10341 raft_consensus.cc:603] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:08.483456 12151 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.589838 12050 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:08.579185 10340 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.592026 10340 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.593726 12340 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:08.597735 11852 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:08.598296 12340 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:08.598695 12340 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
W20240507 08:06:08.599589 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 69354.8us
I20240507 08:06:08.600572 10231 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.603341 12312 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:08.603962 12312 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240507 08:06:08.605707 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 73033.1us
I20240507 08:06:08.606915 12373 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:08.607132 12151 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.607283 12373 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:08.607646 12373 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:08.609668 11754 raft_consensus.cc:979] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:08.609462 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.611907 10336 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:08.612593 10188 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:08.613078 12050 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:08.613243 10188 raft_consensus.cc:595] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:08.614063 12050 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.614854 12284 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:08.616187 11754 raft_consensus.cc:1067] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:08.617902 12379 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.610940 10340 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:08.639024 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:08.642311 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:08.654757 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 114903us
I20240507 08:06:08.661993 10332 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:08.669867 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.670794 10332 raft_consensus.cc:2379] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:08.673056 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:08.674746 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:08.675472 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:08.676718 11654 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:08.678017 12245 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:08.678407 12245 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:08.679147 12245 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.679227 11654 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:08.680007 12245 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:08.684784 10089 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 3 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 5 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:08.687027 10089 catalog_manager.cc:5484] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 1 to 3, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 3 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:08.713361 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:08.654381 12284 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:08.723588 12311 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:08.724138 12311 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:08.750018 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 228963us
I20240507 08:06:08.755403 10266 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:08.759194 10260 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:08.783514 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.784204 10334 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:08.785159 10335 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:08.791545 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:08.786134 10266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "324b71380c6c47bbb335661d299baea0"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:08.794864 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.807304 11866 consensus_queue.cc:1046] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:08.794258 10266 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:08.812132 10266 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:08.819358 11973 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:08.819741 11973 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:08.820508 11973 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.827277 11973 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.830924 12373 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.834352 12375 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:08.834903 12375 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:08.835297 12375 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.834236 12340 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.837267 10192 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:08.837702 12376 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20240507 08:06:08.860543 12334 log.cc:924] Time spent T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:06:08.867913 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.872872 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:08.843006 10266 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.881644 10266 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.883751 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.884465 10335 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:08.887799 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:08.891644 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:08.892439 10192 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:08.892823 10192 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:08.900048 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:08.900769 10337 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:08.808379 11654 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20240507 08:06:08.893832 10192 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.905385 10192 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:08.931353 10192 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:08.882020 12132 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:08.935231 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:08.945017 10337 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:08.902024 12373 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:08.947526 12354 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:08.955686 12354 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:08.957140 10088 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59948 (request call id 164) took 10473 ms. Trace:
I20240507 08:06:08.958228 12354 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:08.957510 10088 rpcz_store.cc:276] 0507 08:05:58.483863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:05:58.484187 (+ 324us) service_pool.cc:225] Handling call
0507 08:06:08.957068 (+10472881us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:04.533764 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:04.534378 (+ 614us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:04.534430 (+ 52us) write_op.cc:435] Acquired schema lock
0507 08:06:04.534464 (+ 34us) tablet.cc:660] Decoding operations
0507 08:06:04.535533 (+ 1069us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:04.535633 (+ 100us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:04.535662 (+ 29us) tablet.cc:683] Acquiring locks for 10 operations
0507 08:06:04.536538 (+ 876us) tablet.cc:699] Row locks acquired
0507 08:06:04.536570 (+ 32us) write_op.cc:260] PREPARE: finished
0507 08:06:04.536739 (+ 169us) write_op.cc:270] Start()
0507 08:06:04.536881 (+ 142us) write_op.cc:276] Timestamp: P: 1715069164536672 usec, L: 0
0507 08:06:04.536915 (+ 34us) op_driver.cc:331] REPLICATION: starting
0507 08:06:04.577048 (+ 40133us) log.cc:842] Serialized 4086 byte log entry
0507 08:06:04.834651 (+257603us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf8dedf16e 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb
0507 08:06:06.051266 (+1216615us) op_driver.cc:444] REPLICATION: finished
0507 08:06:07.673358 (+1622092us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf8dedf16e 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf94838dd0
0507 08:06:07.820941 (+147583us) write_op.cc:301] APPLY: starting
0507 08:06:07.821048 (+ 107us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:07.821269 (+ 221us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:07.821302 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:07.868876 (+ 47574us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:07.869180 (+ 304us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0507 08:06:07.876415 (+ 7235us) write_op.cc:312] APPLY: finished
0507 08:06:07.876791 (+ 376us) log.cc:842] Serialized 76 byte log entry
0507 08:06:07.943800 (+ 67009us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:07.943993 (+ 193us) write_op.cc:454] Released schema lock
0507 08:06:07.952799 (+ 8806us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:08.010424 (+ 57625us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:08.012410 (+ 1986us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":363957,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":1727805,"apply.run_cpu_time_us":44033,"apply.run_wall_time_us":203556,"num_ops":10,"prepare.queue_time_us":37143,"prepare.run_cpu_time_us":60022,"prepare.run_wall_time_us":956025,"raft.queue_time_us":475892,"raft.run_cpu_time_us":35191,"raft.run_wall_time_us":1809514,"replication_time_us":1508455,"spinlock_wait_cycles":24875008,"thread_start_us":2777946,"threads_started":5,"wal-append.queue_time_us":1050950,"wal-append.run_cpu_time_us":19980,"wal-append.run_wall_time_us":1199347}]]}
I20240507 08:06:08.963704 12060 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:08.964134 12060 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:08.905347 10943 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:08.966388 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:08.967962 10231 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:08.933117 12322 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:08.952210 12379 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:08.983177 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:08.971374 12390 log.cc:924] Time spent T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
I20240507 08:06:08.985451 10158 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:08.986413 10231 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:09.002458 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:09.005553 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:09.011626 11853 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:09.024202 10306 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:09.031299 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:08.989135 12388 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:08.990352 12377 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:08.990340 12379 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:09.012187 11853 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:08.991742 12311 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:09.047307 10335 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:09.048892 12340 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:09.071290 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 47181.8us
I20240507 08:06:09.076669 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:09.079972 10340 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:08.977499 12322 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:09.063309 12388 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:08.992926 12060 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.113422 10340 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:09.104794 12129 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240507 08:06:09.153138 12397 log.cc:924] Time spent T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20240507 08:06:09.139580 11853 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:09.125908 12379 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:09.164196 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 50281.7us
I20240507 08:06:09.180172 10156 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:09.181293 10340 raft_consensus.cc:1226] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:09.185376 12281 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:09.190774 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:09.148639 12388 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.218608 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:09.219260 10943 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:09.219789 12375 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.220966 10264 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:09.222551 12375 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:09.205261 12281 raft_consensus.cc:686] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:09.224838 12281 consensus_queue.cc:237] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.197942 12404 consensus_queue.cc:1046] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20240507 08:06:09.228027 12404 raft_consensus.cc:3032] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:09.228452 12404 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:09.244257 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45479.7us
I20240507 08:06:09.231719 10307 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:09.248636 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:09.249382 10339 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:09.250135 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:09.203536 12322 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:09.229223 12404 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.257032 12404 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:09.257071 10232 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:09.257710 11992 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:09.258221 11992 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:09.258626 11992 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:09.260078 12411 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20240507 08:06:09.294047 12384 log.cc:924] Time spent T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.495s user 0.004s sys 0.000s
W20240507 08:06:09.296521 12401 log.cc:924] Time spent T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.184s user 0.000s sys 0.002s
I20240507 08:06:09.314580 10339 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:09.324649 12399 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.202s user 0.000s sys 0.000s
W20240507 08:06:09.324577 12402 log.cc:924] Time spent T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.208s user 0.002s sys 0.000s
I20240507 08:06:09.383035 12060 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:09.414674 12403 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.340s user 0.002s sys 0.000s
I20240507 08:06:09.411490 11992 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:09.455226 12282 raft_consensus.cc:979] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:09.457340 12387 raft_consensus.cc:1067] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:09.459137 10264 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:09.459647 10264 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:09.460002 10264 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:09.467250 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:09.474103 10264 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.476090 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:09.476500 10191 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:09.476706 10191 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:09.477387 10191 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.481763 10191 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:09.489979 10264 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:09.490823 12418 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.162s user 0.000s sys 0.002s
I20240507 08:06:09.492627 10089 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 70) took 7212 ms. Trace:
I20240507 08:06:09.492974 10089 rpcz_store.cc:276] 0507 08:06:02.280649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.286102 (+ 5453us) service_pool.cc:225] Handling call
0507 08:06:09.492561 (+7206459us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:09.012740 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:09.015692 (+ 2952us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:09.015736 (+ 44us) write_op.cc:435] Acquired schema lock
0507 08:06:09.015753 (+ 17us) tablet.cc:660] Decoding operations
0507 08:06:09.016177 (+ 424us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:09.016263 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:09.016308 (+ 45us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:06:09.016619 (+ 311us) tablet.cc:699] Row locks acquired
0507 08:06:09.016639 (+ 20us) write_op.cc:260] PREPARE: finished
0507 08:06:09.016833 (+ 194us) write_op.cc:270] Start()
0507 08:06:09.016956 (+ 123us) write_op.cc:276] Timestamp: P: 1715069169016767 usec, L: 0
0507 08:06:09.016982 (+ 26us) op_driver.cc:331] REPLICATION: starting
0507 08:06:09.044360 (+ 27378us) log.cc:842] Serialized 918 byte log entry
0507 08:06:09.163034 (+118674us) op_driver.cc:444] REPLICATION: finished
0507 08:06:09.293754 (+130720us) write_op.cc:301] APPLY: starting
0507 08:06:09.293844 (+ 90us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:09.293977 (+ 133us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:09.294100 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:09.294571 (+ 471us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:09.294730 (+ 159us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0507 08:06:09.294775 (+ 45us) write_op.cc:312] APPLY: finished
0507 08:06:09.294969 (+ 194us) log.cc:842] Serialized 28 byte log entry
0507 08:06:09.296692 (+ 1723us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:09.297083 (+ 391us) write_op.cc:454] Released schema lock
0507 08:06:09.297278 (+ 195us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:09.298650 (+ 1372us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:09.302206 (+ 3556us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":442810,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":121759,"apply.run_cpu_time_us":2187,"apply.run_wall_time_us":9817,"num_ops":2,"prepare.queue_time_us":289834,"prepare.run_cpu_time_us":3076,"prepare.run_wall_time_us":67974,"replication_time_us":145927,"thread_start_us":263797,"threads_started":2}]]}
I20240507 08:06:09.495805 10090 catalog_manager.cc:5484] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 4, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.499081 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:09.499648 10333 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:09.501921 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:09.502504 10943 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:09.503438 10191 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:09.505945 10232 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:09.506453 10333 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:09.506690 12245 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:09.507222 12245 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:09.507989 12245 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.497284 10090 catalog_manager.cc:5484] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.517686 10943 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:09.519286 12386 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:09.519881 12386 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.520278 10307 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:09.521152 12331 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:09.521641 12331 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:09.521924 12386 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:09.509080 11992 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:09.522517 12331 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:09.523505 10090 catalog_manager.cc:5484] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 2, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 2 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.524030 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:09.524825 10260 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:09.526564 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:09.527031 10333 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:09.527804 10156 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:09.528548 10090 catalog_manager.cc:5484] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.530083 10090 catalog_manager.cc:5484] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.529155 12282 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:09.531852 10090 catalog_manager.cc:5484] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.532622 12282 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:09.533179 12282 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:09.533313 10090 catalog_manager.cc:5484] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.535580 10090 catalog_manager.cc:5484] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 6, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.547179 12282 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.550012 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:09.557412 10264 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:09.563061 12282 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:09.536852 10090 catalog_manager.cc:5484] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.569617 10090 catalog_manager.cc:5484] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.572425 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:09.574819 10090 catalog_manager.cc:5484] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.575875 10090 catalog_manager.cc:5484] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.588541 10090 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.590435 10090 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.591959 10090 catalog_manager.cc:5484] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 4, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.600575 10333 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:09.601137 10090 catalog_manager.cc:5484] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.610585 10090 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 2, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 2 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.612306 10090 catalog_manager.cc:5484] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.614073 10156 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:09.615217 11853 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:09.615700 11853 raft_consensus.cc:686] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:09.615957 10090 catalog_manager.cc:5484] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.616539 11853 consensus_queue.cc:237] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.617890 10090 catalog_manager.cc:5484] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.630702 10090 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 4, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.632212 10090 catalog_manager.cc:5484] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.634161 10090 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.635452 10090 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 4, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.693533 10333 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:09.694190 10334 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:09.695722 12381 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:09.695760 12245 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:09.711463 10089 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.716336 10340 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:09.729820 10089 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 4 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:09.771988 10191 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240507 08:06:09.774215 10191 pending_rounds.cc:76] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 2
I20240507 08:06:09.774542 10191 pending_rounds.cc:97] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240507 08:06:09.774868 10191 raft_consensus.cc:2866] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:09.776330 12381 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:09.778215 10260 raft_consensus.cc:1261] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:09.779673 12404 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:09.800935 10185 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:09.806823 12381 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240507 08:06:09.846330 10307 consensus_peers.cc:485] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffd4c65b5ad04f48916c3e359f7e45c9. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:09.853443 10264 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:09.855232 12355 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:09.862438 10334 raft_consensus.cc:1261] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:09.867697 12404 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:09.876930 10185 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:09.879212 12355 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:09.881196 12331 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20240507 08:06:09.895979 10090 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59954 (request call id 160) took 7537 ms. Trace:
I20240507 08:06:09.885787 12355 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.907625 10086 catalog_manager.cc:5484] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.896353 10090 rpcz_store.cc:276] 0507 08:06:02.358175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:02.367078 (+ 8903us) service_pool.cc:225] Handling call
0507 08:06:09.895924 (+7528846us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:09.682380 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:09.682918 (+ 538us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:09.682957 (+ 39us) write_op.cc:435] Acquired schema lock
0507 08:06:09.682977 (+ 20us) tablet.cc:660] Decoding operations
0507 08:06:09.684918 (+ 1941us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:09.684997 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:09.685035 (+ 38us) tablet.cc:683] Acquiring locks for 24 operations
0507 08:06:09.687140 (+ 2105us) tablet.cc:699] Row locks acquired
0507 08:06:09.687158 (+ 18us) write_op.cc:260] PREPARE: finished
0507 08:06:09.687310 (+ 152us) write_op.cc:270] Start()
0507 08:06:09.687426 (+ 116us) write_op.cc:276] Timestamp: P: 1715069169687260 usec, L: 0
0507 08:06:09.687444 (+ 18us) op_driver.cc:331] REPLICATION: starting
0507 08:06:09.688121 (+ 677us) log.cc:842] Serialized 9457 byte log entry
0507 08:06:09.691253 (+ 3132us) op_driver.cc:444] REPLICATION: finished
0507 08:06:09.691692 (+ 439us) write_op.cc:301] APPLY: starting
0507 08:06:09.691762 (+ 70us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:09.692181 (+ 419us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:09.692279 (+ 98us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:09.696459 (+ 4180us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:09.696755 (+ 296us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=24
0507 08:06:09.696798 (+ 43us) write_op.cc:312] APPLY: finished
0507 08:06:09.697169 (+ 371us) log.cc:842] Serialized 163 byte log entry
0507 08:06:09.697311 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:09.697491 (+ 180us) write_op.cc:454] Released schema lock
0507 08:06:09.697545 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:09.697806 (+ 261us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:09.697832 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13156,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":226,"apply.run_cpu_time_us":4770,"apply.run_wall_time_us":6704,"num_ops":24,"prepare.queue_time_us":12831,"prepare.run_cpu_time_us":6015,"prepare.run_wall_time_us":6015,"replication_time_us":3727,"thread_start_us":13360,"threads_started":1}]]}
I20240507 08:06:09.915760 10086 catalog_manager.cc:5484] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.919298 10086 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.922708 10339 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:09.922863 10086 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.924227 10086 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.925784 10086 catalog_manager.cc:5484] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.929126 10086 catalog_manager.cc:5484] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 5, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.932413 10086 catalog_manager.cc:5484] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.934070 10086 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.935740 10086 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 5, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.939443 10086 catalog_manager.cc:5484] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:09.940757 10086 catalog_manager.cc:5484] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 5 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.942137 10086 catalog_manager.cc:5484] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.943454 10086 catalog_manager.cc:5484] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.943308 10943 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:09.946547 10086 catalog_manager.cc:5484] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 2 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.934074 12381 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20240507 08:06:09.954514 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:09.957542 10086 catalog_manager.cc:5484] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.956451 11992 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:09.966491 10086 catalog_manager.cc:5484] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.963869 12060 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:09.976418 10086 catalog_manager.cc:5484] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:09.977470 10086 catalog_manager.cc:5484] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.979923 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:09.978967 10086 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:09.988086 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:09.988966 10263 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:09.992241 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:09.976029 12060 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:09.994182 12060 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:09.996048 12331 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:09.996461 12331 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:09.996809 12331 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:09.982573 11992 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:09.998286 11992 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.000631 11992 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.011564 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:09.982565 12355 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.015833 10188 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:10.012231 10191 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:10.017774 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:10.020792 12436 log.cc:924] Time spent T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20240507 08:06:10.023725 12060 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:10.024082 12060 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:10.024402 12060 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:10.028096 12312 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:10.070850 10339 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:10.049508 12060 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:10.076120 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 83972.4us
I20240507 08:06:10.077322 10332 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:10.077759 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:10.077879 10332 raft_consensus.cc:595] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:10.085729 10263 raft_consensus.cc:2379] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:10.088044 12331 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.078841 11992 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:10.096175 11992 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.098299 11992 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.100817 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:10.101482 10339 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:10.080425 12404 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20240507 08:06:10.118700 12060 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.120349 12404 consensus_queue.cc:1225] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer 88a26dac37fa4243b8c2b3d47f403488 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240507 08:06:10.120940 12386 raft_consensus.cc:3032] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:10.121294 12386 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:10.122167 12331 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.122071 12386 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:10.154747 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:10.155568 10191 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:10.159984 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:10.160777 10339 raft_consensus.cc:595] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:10.161211 10339 raft_consensus.cc:603] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:10.181907 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:10.183053 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:10.183687 10339 raft_consensus.cc:595] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:10.184301 10339 raft_consensus.cc:603] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:10.179738 10191 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:10.137969 12256 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:10.187885 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:10.191829 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:10.203394 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:10.203958 10339 raft_consensus.cc:595] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:10.204305 10339 raft_consensus.cc:603] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:10.207722 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:10.208262 10339 raft_consensus.cc:595] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:10.232753 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:10.234930 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:10.235790 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:10.248919 12386 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:10.261960 10185 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20240507 08:06:10.262346 10185 pending_rounds.cc:76] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 7
I20240507 08:06:10.262557 10185 pending_rounds.cc:97] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20240507 08:06:10.262776 10185 raft_consensus.cc:2866] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:10.280350 10090 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 6 to 8, leader changed from 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 8 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:10.283826 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:10.284868 12245 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:10.285470 12245 raft_consensus.cc:686] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:10.286511 12245 consensus_queue.cc:237] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:10.292451 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 163676us
W20240507 08:06:10.303869 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 85515.5us
I20240507 08:06:10.305503 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:10.306170 10263 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:10.306191 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:10.311241 10260 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:10.318153 10264 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:10.318907 10304 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:10.320719 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:10.321298 10332 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:10.322088 10185 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:10.311036 12447 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:10.324537 12446 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:10.332675 12445 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:10.332553 12450 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20240507 08:06:10.343480 12447 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.400903 12447 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.351298 12440 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:10.401691 12440 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.403537 12440 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.375634 12446 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.407338 12446 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.366323 12445 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.409417 12445 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:10.410446 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 91664.4us
W20240507 08:06:10.412611 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 99463.8us
I20240507 08:06:10.417443 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:10.418144 10260 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:10.421193 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:10.421809 10188 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:10.423578 10306 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:10.426314 10258 raft_consensus.cc:2379] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:10.427050 10304 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:10.433823 10332 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:10.439612 12454 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:10.440008 12454 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:10.440337 12454 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:10.434648 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 112910us
I20240507 08:06:10.444604 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:10.446508 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:10.447362 10266 raft_consensus.cc:1261] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:10.447297 10265 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:10.444224 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:10.450217 10258 raft_consensus.cc:2354] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.445333 10260 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:10.456840 10340 raft_consensus.cc:1261] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20240507 08:06:10.469233 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 45749.9us
I20240507 08:06:10.470254 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:10.470786 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:10.471066 10340 raft_consensus.cc:2354] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.473515 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:10.474156 10339 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:10.441805 12331 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240507 08:06:10.474496 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 51367us
I20240507 08:06:10.510957 10184 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:10.484097 12256 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20240507 08:06:10.530589 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47421.8us
W20240507 08:06:10.531968 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 46783us
I20240507 08:06:10.539553 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:10.536523 10940 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:10.530403 12256 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:10.555600 12457 ts_tablet_manager.cc:902] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.568497 12454 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.575526 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:10.581136 10334 raft_consensus.cc:2354] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.582243 12454 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.571120 12355 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:10.574164 12256 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:10.599593 10155 leader_election.cc:398] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.599993 10155 leader_election.cc:401] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:10.601162 12355 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:10.602306 12355 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.603535 11924 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:10.604022 11924 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:10.604420 11924 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:10.586867 12460 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20240507 08:06:10.633798 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 81383.1us
W20240507 08:06:10.647754 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 57621.8us
I20240507 08:06:10.648885 10156 leader_election.cc:398] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.655643 12449 raft_consensus.cc:979] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:10.655892 12376 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:10.656210 12449 raft_consensus.cc:1067] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:10.656597 12376 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.659291 12376 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:10.663583 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 64173us
I20240507 08:06:10.664359 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:10.665755 12404 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:10.668560 11754 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 7
I20240507 08:06:10.668596 12464 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:10.670360 12464 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:10.670702 12464 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:10.676101 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:10.679279 12256 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.688938 12464 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.697916 12256 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:10.701646 12464 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.703224 10156 leader_election.cc:398] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.703687 10086 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59936 (request call id 157) took 2517 ms. Trace:
I20240507 08:06:10.705777 11754 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.622676 11924 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.709018 11924 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.709619 10088 catalog_manager.cc:5484] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 4 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.716845 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:10.718106 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:10.726167 10260 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:10.704077 10086 rpcz_store.cc:276] 0507 08:06:08.186137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:08.186370 (+ 233us) service_pool.cc:225] Handling call
0507 08:06:10.703606 (+2517236us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:10.174617 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:10.175091 (+ 474us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:10.175131 (+ 40us) write_op.cc:435] Acquired schema lock
0507 08:06:10.175150 (+ 19us) tablet.cc:660] Decoding operations
0507 08:06:10.176720 (+ 1570us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:10.176774 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:10.176801 (+ 27us) tablet.cc:683] Acquiring locks for 20 operations
0507 08:06:10.248813 (+ 72012us) tablet.cc:699] Row locks acquired
0507 08:06:10.248836 (+ 23us) write_op.cc:260] PREPARE: finished
0507 08:06:10.248980 (+ 144us) write_op.cc:270] Start()
0507 08:06:10.249112 (+ 132us) write_op.cc:276] Timestamp: P: 1715069170248935 usec, L: 0
0507 08:06:10.249134 (+ 22us) op_driver.cc:331] REPLICATION: starting
0507 08:06:10.249792 (+ 658us) log.cc:842] Serialized 7913 byte log entry
0507 08:06:10.253310 (+ 3518us) op_driver.cc:444] REPLICATION: finished
0507 08:06:10.253900 (+ 590us) write_op.cc:301] APPLY: starting
0507 08:06:10.253992 (+ 92us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:10.254511 (+ 519us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:10.254620 (+ 109us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:10.256874 (+ 2254us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:10.258586 (+ 1712us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=20
0507 08:06:10.258704 (+ 118us) write_op.cc:312] APPLY: finished
0507 08:06:10.259208 (+ 504us) log.cc:842] Serialized 138 byte log entry
0507 08:06:10.259381 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:10.259610 (+ 229us) write_op.cc:454] Released schema lock
0507 08:06:10.259681 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:10.259959 (+ 278us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:10.260756 (+ 797us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":523289,"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":360,"apply.run_cpu_time_us":5324,"apply.run_wall_time_us":7943,"num_ops":20,"prepare.queue_time_us":125,"prepare.run_cpu_time_us":5380,"prepare.run_wall_time_us":81476,"replication_time_us":4076}]]}
I20240507 08:06:10.732847 10261 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:10.739579 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:10.740278 10332 raft_consensus.cc:2354] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.741955 12457 tablet_copy_client.cc:316] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.131:37047
I20240507 08:06:10.738916 10088 catalog_manager.cc:5484] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:10.750361 10088 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:10.756642 10351 tablet_copy_service.cc:140] P 88a26dac37fa4243b8c2b3d47f403488: Received BeginTabletCopySession request for tablet ffd4c65b5ad04f48916c3e359f7e45c9 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:47396)
I20240507 08:06:10.761056 10351 tablet_copy_service.cc:161] P 88a26dac37fa4243b8c2b3d47f403488: Beginning new tablet copy session on tablet ffd4c65b5ad04f48916c3e359f7e45c9 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:47396: session id = 5d6656a616b747d393ff3de5328d2540-ffd4c65b5ad04f48916c3e359f7e45c9
I20240507 08:06:10.769148 10351 tablet_copy_source_session.cc:205] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:10.748378 10260 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:10.773914 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:10.774747 10340 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:10.773123 10088 catalog_manager.cc:5484] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.785260 10088 catalog_manager.cc:5484] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:10.789937 10340 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:10.799611 12457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd4c65b5ad04f48916c3e359f7e45c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:10.800637 10088 catalog_manager.cc:5484] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.803050 10088 catalog_manager.cc:5484] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
W20240507 08:06:10.819411 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 54726.9us
I20240507 08:06:10.819891 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:10.820485 10183 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:10.824766 10191 raft_consensus.cc:1261] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:10.804704 10088 catalog_manager.cc:5484] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 1 to 3, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:10.826778 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:10.827762 10088 catalog_manager.cc:5484] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 4 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.829641 10185 raft_consensus.cc:2354] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6.
I20240507 08:06:10.829288 10088 catalog_manager.cc:5484] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 6, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
W20240507 08:06:10.842428 10158 consensus_peers.cc:485] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e940ba6acb8d4782a4c921b50b882f91. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:10.842161 10088 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.846489 10088 catalog_manager.cc:5484] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.833842 12376 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:10.835407 10306 leader_election.cc:398] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6.
I20240507 08:06:10.848953 10306 leader_election.cc:401] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240507 08:06:10.837106 10183 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
W20240507 08:06:10.830879 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 276724us
I20240507 08:06:10.850335 10306 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:10.852250 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:10.852910 10261 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:10.855405 10156 leader_election.cc:398] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.855825 10156 leader_election.cc:401] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240507 08:06:10.856775 10264 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42924
I20240507 08:06:10.857363 10264 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:10.857729 10264 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:10.848169 10088 catalog_manager.cc:5484] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:10.861145 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:10.863751 10266 raft_consensus.cc:2354] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6.
I20240507 08:06:10.868237 10264 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.869906 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:10.871490 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:10.872027 10183 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:10.874433 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:10.876099 10264 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:10.876442 10266 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:10.878648 12378 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:10.863289 12440 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:10.882344 12440 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:10.860653 12465 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6.
I20240507 08:06:10.866565 11924 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:10.886999 10261 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:10.888289 10304 leader_election.cc:398] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6.
I20240507 08:06:10.891927 10334 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:10.888156 10183 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:10.894034 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:10.889209 12378 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:10.902879 12378 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.908447 12443 raft_consensus.cc:979] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:10.914819 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:10.915393 10332 raft_consensus.cc:3032] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:10.915771 10332 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:10.883190 12440 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.913745 11941 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20240507 08:06:10.919875 10332 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.923758 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:10.918740 12060 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:10.941004 10332 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:10.920219 12443 raft_consensus.cc:1067] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:10.961391 12378 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240507 08:06:10.969797 10340 raft_consensus.cc:1261] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:10.939872 12312 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:10.959944 12060 raft_consensus.cc:686] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:10.977988 12060 consensus_queue.cc:237] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:10.985222 10332 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:10.987833 10261 raft_consensus.cc:1226] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:10.995817 12060 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:11.005748 10264 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42924
I20240507 08:06:11.006224 10264 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:11.006544 10264 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:11.023391 10264 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:11.026160 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:11.026786 10185 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:11.033250 10185 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:11.037784 10232 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:11.043098 10264 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:11.044131 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:11.044899 10339 raft_consensus.cc:3032] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:11.045254 10339 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:11.067584 12462 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.042776 12060 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:11.071899 12060 raft_consensus.cc:686] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:11.071069 10260 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.072784 12060 consensus_queue.cc:237] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:11.076233 10339 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:11.077438 10339 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:11.019397 12457 tablet_copy_client.cc:791] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:11.112504 10339 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:11.118563 12457 tablet_copy_client.cc:655] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:11.137081 10264 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:11.138537 12443 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:11.156800 10332 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.149081 12457 tablet_copy_client.cc:531] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:11.195079 10185 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:11.196791 12465 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:11.202560 10339 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:11.227061 10183 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20240507 08:06:11.228729 10307 consensus_peers.cc:485] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85b6fa5360054f32af996ec68d175584. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:11.231701 12465 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:11.237237 10261 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:11.248567 12443 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:11.251924 12457 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:11.293480 12256 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.308473 10332 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:11.310186 12461 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:11.316255 10185 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:11.313601 10265 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.324070 10263 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:11.325413 10086 catalog_manager.cc:5484] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:11.328758 12450 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:11.333616 10088 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59948 (request call id 165) took 1805 ms. Trace:
I20240507 08:06:11.333171 10086 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:11.347069 10183 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.333957 10088 rpcz_store.cc:276] 0507 08:06:09.526378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:09.526564 (+ 186us) service_pool.cc:225] Handling call
0507 08:06:11.332086 (+1805522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:10.989883 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:10.990420 (+ 537us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:10.990456 (+ 36us) write_op.cc:435] Acquired schema lock
0507 08:06:10.990477 (+ 21us) tablet.cc:660] Decoding operations
0507 08:06:10.991650 (+ 1173us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:10.991710 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:10.991740 (+ 30us) tablet.cc:683] Acquiring locks for 13 operations
0507 08:06:10.992893 (+ 1153us) tablet.cc:699] Row locks acquired
0507 08:06:10.992914 (+ 21us) write_op.cc:260] PREPARE: finished
0507 08:06:10.993067 (+ 153us) write_op.cc:270] Start()
0507 08:06:10.993175 (+ 108us) write_op.cc:276] Timestamp: P: 1715069170993009 usec, L: 0
0507 08:06:10.993199 (+ 24us) op_driver.cc:331] REPLICATION: starting
0507 08:06:11.011814 (+ 18615us) log.cc:842] Serialized 5193 byte log entry
0507 08:06:11.132579 (+120765us) op_driver.cc:444] REPLICATION: finished
0507 08:06:11.224098 (+ 91519us) write_op.cc:301] APPLY: starting
0507 08:06:11.224957 (+ 859us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:11.245088 (+ 20131us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:11.245257 (+ 169us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:11.246367 (+ 1110us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:11.259972 (+ 13605us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=13
0507 08:06:11.260019 (+ 47us) write_op.cc:312] APPLY: finished
0507 08:06:11.260312 (+ 293us) log.cc:842] Serialized 94 byte log entry
0507 08:06:11.260469 (+ 157us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:11.260607 (+ 138us) write_op.cc:454] Released schema lock
0507 08:06:11.260655 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:11.260876 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:11.260906 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4314,"reported_tablets":53,"child_traces":[["op",{"apply.queue_time_us":69649,"apply.run_cpu_time_us":3409,"apply.run_wall_time_us":37913,"num_ops":13,"prepare.queue_time_us":115099,"prepare.run_cpu_time_us":4421,"prepare.run_wall_time_us":23960,"replication_time_us":133997,"thread_start_us":121184,"threads_started":2}]]}
I20240507 08:06:11.341311 10086 catalog_manager.cc:5484] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:11.404937 10086 catalog_manager.cc:5484] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 7, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:11.422658 12457 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:11.423631 12457 tablet_bootstrap.cc:492] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:11.424329 12457 ts_tablet_manager.cc:1367] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.173s user 0.026s sys 0.000s
I20240507 08:06:11.463213 10183 raft_consensus.cc:1261] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:11.490514 10184 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:11.492131 12440 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:11.497519 10335 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:11.498876 12381 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:11.500156 12381 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:11.508430 10260 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:11.510275 10339 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240507 08:06:11.512964 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 110783us
I20240507 08:06:11.517347 10090 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:11.531358 12464 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:11.532377 11941 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:11.535924 10090 catalog_manager.cc:5484] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:11.548368 12457 raft_consensus.cc:348] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:11.549284 12457 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:11.550978 10090 catalog_manager.cc:5484] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 1 to 5, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:11.550045 12457 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:11.570835 12504 ts_tablet_manager.cc:902] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:06:11.571476 10263 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20240507 08:06:11.583865 10158 consensus_peers.cc:485] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9800706282e543be9f2ee3473da2b89f. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:11.595983 10184 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:11.586617 12457 ts_tablet_manager.cc:1398] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.162s user 0.004s sys 0.008s
I20240507 08:06:11.597312 12312 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:11.595510 12445 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20240507 08:06:11.611389 12446 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.616484 10351 tablet_copy_service.cc:342] P 88a26dac37fa4243b8c2b3d47f403488: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-ffd4c65b5ad04f48916c3e359f7e45c9 received from {username='slave'} at 127.0.0.1:47396
I20240507 08:06:11.616899 10351 tablet_copy_service.cc:433] P 88a26dac37fa4243b8c2b3d47f403488: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-ffd4c65b5ad04f48916c3e359f7e45c9 on tablet ffd4c65b5ad04f48916c3e359f7e45c9 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:11.600370 10088 catalog_manager.cc:5484] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.624276 10335 raft_consensus.cc:1261] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:11.625422 10088 catalog_manager.cc:5484] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:11.627071 10088 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 7, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:11.629063 10088 catalog_manager.cc:5484] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 4, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:11.630880 12312 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:11.626292 12504 tablet_copy_client.cc:316] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:06:11.629086 10340 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.640789 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55810
I20240507 08:06:11.654325 10185 raft_consensus.cc:595] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership
I20240507 08:06:11.655577 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet e940ba6acb8d4782a4c921b50b882f91 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:06:11.656116 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet e940ba6acb8d4782a4c921b50b882f91 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-e940ba6acb8d4782a4c921b50b882f91
I20240507 08:06:11.659837 10088 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.693085 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:11.693675 12457 ts_tablet_manager.cc:902] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:11.694705 10339 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:11.718595 10202 tablet_copy_source_session.cc:205] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:11.735162 10940 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:11.735029 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55810
I20240507 08:06:11.738655 10184 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:11.763041 10185 raft_consensus.cc:595] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership
I20240507 08:06:11.742890 12457 tablet_copy_client.cc:316] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.131:37047
W20240507 08:06:11.766076 10076 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for e940ba6acb8d4782a4c921b50b882f91 already in progress [suppressed 5 similar messages]
I20240507 08:06:11.766346 12440 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:11.768079 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:11.768666 10185 raft_consensus.cc:595] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:11.749480 10266 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:11.773303 10351 tablet_copy_service.cc:140] P 88a26dac37fa4243b8c2b3d47f403488: Received BeginTabletCopySession request for tablet 85b6fa5360054f32af996ec68d175584 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:47396)
I20240507 08:06:11.773857 10351 tablet_copy_service.cc:161] P 88a26dac37fa4243b8c2b3d47f403488: Beginning new tablet copy session on tablet 85b6fa5360054f32af996ec68d175584 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:47396: session id = 5d6656a616b747d393ff3de5328d2540-85b6fa5360054f32af996ec68d175584
I20240507 08:06:11.774518 12450 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:11.775076 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940ba6acb8d4782a4c921b50b882f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:11.790478 10939 raft_consensus.cc:1203] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.3->[4.4-4.4] Dedup: 4.4->[]
I20240507 08:06:11.792115 10184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:11.792675 10184 raft_consensus.cc:595] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:11.798843 12446 raft_consensus.cc:979] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:11.801595 12446 raft_consensus.cc:1067] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:11.803115 10351 tablet_copy_source_session.cc:205] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:11.817898 10340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:11.818329 10340 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:11.818593 10340 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:11.820135 10090 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:11.832984 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:11.833544 10185 raft_consensus.cc:595] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:11.834455 10340 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:11.836438 10340 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:11.836815 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:11.837406 10192 raft_consensus.cc:3032] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:11.837711 10192 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:11.848579 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:11.849248 10266 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:11.856204 10266 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:11.839716 10192 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:11.858867 10192 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:11.859602 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:11.860157 10185 raft_consensus.cc:595] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:11.860265 10304 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:11.868204 10184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:11.868757 10184 raft_consensus.cc:595] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:11.869813 12446 raft_consensus.cc:979] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:11.870229 12446 raft_consensus.cc:1067] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:11.870361 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:11.871755 10340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:11.872234 10340 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:11.872542 10340 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:11.874578 12462 raft_consensus.cc:979] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:11.875177 12462 raft_consensus.cc:1067] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:11.876271 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:11.877408 12457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b6fa5360054f32af996ec68d175584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:11.876933 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:11.883723 10340 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:11.886981 10340 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:11.891175 10260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:11.891657 10260 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:11.891971 10260 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
W20240507 08:06:11.906699 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 46836.9us
W20240507 08:06:11.928870 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 96564us
I20240507 08:06:11.936002 10260 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:11.938155 10260 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:11.945405 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:11.945850 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 59993us
I20240507 08:06:11.946015 10184 raft_consensus.cc:3032] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:11.946470 10184 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:11.947901 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:11.948446 10263 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:11.948467 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:11.949469 10183 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:11.949730 10183 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:11.950498 10192 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:11.947134 10184 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:11.954818 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:11.955303 10184 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:11.955348 10340 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:11.958184 12465 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:11.950383 10183 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:11.963868 10183 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:11.970918 10183 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:11.972201 10263 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:11.973608 10183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55810
I20240507 08:06:11.974859 10183 raft_consensus.cc:595] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Received request to transfer leadership
I20240507 08:06:11.975178 10183 raft_consensus.cc:603] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:11.969909 10340 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
W20240507 08:06:11.959973 12516 log.cc:924] Time spent T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.081s user 0.003s sys 0.000s
W20240507 08:06:11.987149 10138 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
W20240507 08:06:11.987447 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20240507 08:06:11.963631 12465 raft_consensus.cc:686] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:12.018435 10184 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
W20240507 08:06:12.022697 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 74517.5us
I20240507 08:06:12.035993 10304 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:12.036976 10232 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:12.044390 12443 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:12.044896 12443 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:11.998555 10940 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 4 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.012739 12465 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:12.051090 10183 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Committing config change with OpId 6.7: config changed from index 4 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.052760 12443 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:11.932590 12440 raft_consensus.cc:2932] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Committing config change with OpId 6.7: config changed from index 4 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.060133 12464 raft_consensus.cc:979] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:12.060616 12464 raft_consensus.cc:1067] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:12.068464 10335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:12.068974 10335 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:12.069296 10335 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:12.089161 12446 raft_consensus.cc:979] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:12.089843 12446 raft_consensus.cc:1067] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:12.092664 10266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:12.093179 10266 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:12.093505 10266 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:12.093828 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 56237.8us
I20240507 08:06:12.086912 10335 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.100685 10266 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.102541 10266 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:12.106436 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:12.107023 10183 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:12.107282 10183 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:12.107926 10183 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.108916 10183 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:12.110700 10335 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:12.112025 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:12.112612 10340 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:12.112046 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:12.113806 10265 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:12.120208 10265 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:12.127452 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:12.128072 10184 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:12.128321 10184 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:12.128973 10184 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.040254 12480 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:12.130060 10184 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:12.130163 12480 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:12.131072 12480 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.132591 10088 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 4 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.138935 12457 tablet_copy_client.cc:791] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:12.139444 10184 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:12.140524 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:12.139480 12457 tablet_copy_client.cc:655] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:12.141284 12465 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:12.142005 12465 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:12.141961 10183 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:12.145119 10232 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:12.147104 10340 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:12.143419 12465 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.147523 12381 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:12.158490 12528 ts_tablet_manager.cc:902] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:06:12.158888 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84da12b367b046b7bcfb08aaa6e1c97b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:12.160243 12504 tablet_copy_client.cc:791] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:12.172315 12381 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:12.176218 12457 tablet_copy_client.cc:531] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:12.177083 12381 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:12.171416 12446 raft_consensus.cc:979] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:12.183831 12446 raft_consensus.cc:1067] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:12.185277 10335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:12.185837 10335 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:12.186183 10335 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:12.188704 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet 84da12b367b046b7bcfb08aaa6e1c97b with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42910
I20240507 08:06:12.194940 12528 tablet_copy_client.cc:316] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:06:12.197098 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet 9800706282e543be9f2ee3473da2b89f from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:06:12.197619 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet 9800706282e543be9f2ee3473da2b89f from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-9800706282e543be9f2ee3473da2b89f
I20240507 08:06:12.189385 10090 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:12.202694 10335 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:12.204568 10335 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:12.173823 12504 tablet_copy_client.cc:655] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:12.216746 10202 tablet_copy_source_session.cc:205] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:12.219148 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:12.219739 10183 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:12.220011 10183 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:12.223213 12504 tablet_copy_client.cc:531] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:12.217415 12457 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:12.229745 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:12.225901 10183 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:12.230309 10263 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:12.231007 10183 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:12.240654 10263 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:12.246188 10183 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:12.247169 10306 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:12.251950 10088 catalog_manager.cc:5484] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:12.254254 10088 catalog_manager.cc:5484] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:12.257191 10088 catalog_manager.cc:5484] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:12.260883 10090 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 7, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:12.256779 12440 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:12.269425 12528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9800706282e543be9f2ee3473da2b89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:12.269562 12533 tablet_replica.cc:330] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:06:12.271337 12533 raft_consensus.cc:2229] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:12.272073 12533 raft_consensus.cc:2258] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:12.274364 12533 ts_tablet_manager.cc:1875] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:12.276329 12440 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:12.279785 12440 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:12.303905 12443 raft_consensus.cc:1050] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:06:12.306574 12443 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:12.354768 10939 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240507 08:06:12.356362 12440 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:12.350874 10088 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:12.366544 12504 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:12.370750 10263 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240507 08:06:12.373481 12256 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:12.381072 10185 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240507 08:06:12.383338 12443 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20240507 08:06:12.402069 10334 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:12.402616 10334 raft_consensus.cc:595] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership
I20240507 08:06:12.436300 12256 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.437914 10185 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.442088 12528 tablet_copy_client.cc:791] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:12.442648 12528 tablet_copy_client.cc:655] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:12.443194 10335 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:12.443715 10335 raft_consensus.cc:595] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership
I20240507 08:06:12.444990 10263 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.451948 10939 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 4 to 5, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.465822 12528 tablet_copy_client.cc:531] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:12.473512 12533 ts_tablet_manager.cc:1888] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.6
I20240507 08:06:12.506958 10263 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:12.510358 10265 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:12.510658 12457 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:12.511561 12457 tablet_bootstrap.cc:492] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:12.511575 12443 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:12.512142 12465 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:12.512395 12457 ts_tablet_manager.cc:1367] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.295s user 0.038s sys 0.016s
I20240507 08:06:12.515605 12457 raft_consensus.cc:348] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:12.516505 12457 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:12.499917 10088 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 4 to 5, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:06:12.517817 10075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 63683.5us
I20240507 08:06:12.500491 12533 log.cc:1196] T 84da12b367b046b7bcfb08aaa6e1c97b P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/84da12b367b046b7bcfb08aaa6e1c97b
I20240507 08:06:12.517163 12457 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:12.552675 10335 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:12.558027 10334 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:12.562587 12381 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20240507 08:06:12.568281 10185 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:12.568773 10183 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:12.569664 12523 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:12.570216 10184 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:06:12.572821 10939 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:06:12.573590 12465 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:12.574738 12443 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:12.591137 12465 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.594307 10192 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.603026 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet 84da12b367b046b7bcfb08aaa6e1c97b (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:12.604885 10334 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:12.605510 10334 raft_consensus.cc:595] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership
I20240507 08:06:12.605144 12528 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
W20240507 08:06:12.608045 10138 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 9800706282e543be9f2ee3473da2b89f already in progress
W20240507 08:06:12.608433 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 9800706282e543be9f2ee3473da2b89f already in progress
I20240507 08:06:12.611418 10260 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:12.622818 12457 ts_tablet_manager.cc:1398] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.110s user 0.005s sys 0.003s
I20240507 08:06:12.618469 12525 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20240507 08:06:12.665712 10185 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:12.623678 12440 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20240507 08:06:12.696027 12523 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20240507 08:06:12.797607 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 63224.7us
W20240507 08:06:12.820310 12543 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20240507 08:06:12.820530 10939 raft_consensus.cc:1203] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
W20240507 08:06:12.853236 10307 consensus_peers.cc:485] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:12.860733 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffd4c65b5ad04f48916c3e359f7e45c9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:12.869669 10351 tablet_copy_service.cc:342] P 88a26dac37fa4243b8c2b3d47f403488: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-85b6fa5360054f32af996ec68d175584 received from {username='slave'} at 127.0.0.1:47396
I20240507 08:06:12.870095 10351 tablet_copy_service.cc:433] P 88a26dac37fa4243b8c2b3d47f403488: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-85b6fa5360054f32af996ec68d175584 on tablet 85b6fa5360054f32af996ec68d175584 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:12.878690 10939 raft_consensus.cc:2932] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:06:12.886468 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 53876.1us
I20240507 08:06:12.889206 10191 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:12.890178 10340 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:12.900247 10089 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 5 to 6, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:12.914037 10185 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:12.916720 12312 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:12.932173 10261 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:12.924515 12256 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
W20240507 08:06:12.951004 10307 consensus_peers.cc:485] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:12.956197 10185 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:12.961939 12552 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:12.976691 12550 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20240507 08:06:12.996713 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet ffd4c65b5ad04f48916c3e359f7e45c9 with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42910
W20240507 08:06:13.020464 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50252.1us
I20240507 08:06:13.038650 12450 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:13.059499 12533 tablet_replica.cc:330] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:06:13.078406 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:13.109302 12440 raft_consensus.cc:979] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:13.109807 12440 raft_consensus.cc:1067] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:13.129125 12504 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240507 08:06:13.161214 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56174.3us
I20240507 08:06:13.170042 10266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42924
I20240507 08:06:13.181229 10266 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:13.181568 10266 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:13.189533 12504 tablet_bootstrap.cc:492] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:13.193117 12504 ts_tablet_manager.cc:1367] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.827s user 0.036s sys 0.018s
I20240507 08:06:13.195971 12504 raft_consensus.cc:348] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.196816 12504 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:13.190258 12533 raft_consensus.cc:2229] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:13.198459 12533 raft_consensus.cc:2258] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Raft consensus is shut down!
W20240507 08:06:13.207748 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 143918us
I20240507 08:06:13.216269 10266 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.219502 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:13.224767 12533 ts_tablet_manager.cc:1875] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:13.227836 12552 raft_consensus.cc:1050] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:06:13.232710 12584 log.cc:924] Time spent T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240507 08:06:13.234263 10266 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:13.247495 10183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:13.197439 12504 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.258352 10183 raft_consensus.cc:595] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:13.232431 12552 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:13.261857 10191 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:13.262632 10191 raft_consensus.cc:595] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:13.265498 10183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a123878203f45319def3c4246b9fe30"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:13.266201 10183 raft_consensus.cc:595] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:13.290205 10191 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:13.290769 10191 raft_consensus.cc:595] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:13.292397 12583 raft_consensus.cc:979] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:13.292914 12583 raft_consensus.cc:1067] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:13.292773 12504 ts_tablet_manager.cc:1398] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.099s user 0.007s sys 0.000s
I20240507 08:06:13.296566 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:13.301631 10191 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:13.302211 10191 raft_consensus.cc:595] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:13.306354 10202 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-e940ba6acb8d4782a4c921b50b882f91 received from {username='slave'} at 127.0.0.1:50108
I20240507 08:06:13.306732 10202 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-e940ba6acb8d4782a4c921b50b882f91 on tablet e940ba6acb8d4782a4c921b50b882f91 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:13.309770 10263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:13.310349 10263 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:13.310745 10263 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:13.311683 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:13.313621 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:13.314211 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:13.320000 10263 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.322299 10334 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:13.322674 10334 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:13.323556 10334 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.333366 12528 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:13.334300 12528 tablet_bootstrap.cc:492] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:13.334234 10334 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:13.335099 12528 ts_tablet_manager.cc:1367] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.730s user 0.054s sys 0.016s
I20240507 08:06:13.337452 12578 raft_consensus.cc:979] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:13.337996 12568 raft_consensus.cc:1067] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:13.339413 10260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:13.339923 10260 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:13.340332 10260 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:13.342037 10334 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:13.339812 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:13.342818 10191 raft_consensus.cc:3032] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:13.343127 10191 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:13.343189 10940 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEARNER]: Advancing to term 5
I20240507 08:06:13.344003 10231 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:13.343948 10191 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.345110 10191 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:13.346396 10263 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:13.346907 12312 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:13.347551 12312 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:13.350425 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:13.351047 10340 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:13.356357 10191 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:13.356793 10260 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.358484 10232 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:13.360168 10260 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:13.361565 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:13.362121 10334 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:13.365082 10340 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:13.348345 12312 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.371975 10334 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:13.372993 10231 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:13.378882 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:13.380589 10191 raft_consensus.cc:3032] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:13.380883 10191 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:13.388821 10191 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.392001 12528 raft_consensus.cc:348] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.393193 12528 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:13.390519 10191 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:13.384029 12480 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:13.402922 10191 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:13.387172 12579 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:13.410478 12480 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:13.413865 12480 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.409672 12528 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:13.409168 12579 raft_consensus.cc:686] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:13.478040 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 62914us
W20240507 08:06:13.478049 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 59736.2us
I20240507 08:06:13.501650 12579 consensus_queue.cc:237] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.511548 12528 ts_tablet_manager.cc:1398] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.176s user 0.007s sys 0.018s
I20240507 08:06:13.519475 12533 ts_tablet_manager.cc:1888] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.5
I20240507 08:06:13.519965 12533 log.cc:1196] T ffd4c65b5ad04f48916c3e359f7e45c9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/ffd4c65b5ad04f48916c3e359f7e45c9
I20240507 08:06:13.575558 10202 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-9800706282e543be9f2ee3473da2b89f received from {username='slave'} at 127.0.0.1:50108
I20240507 08:06:13.576643 10202 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-9800706282e543be9f2ee3473da2b89f on tablet 9800706282e543be9f2ee3473da2b89f with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:13.591866 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet ffd4c65b5ad04f48916c3e359f7e45c9 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:13.622418 10940 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:13.630640 10191 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:13.631512 10263 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:13.654467 12568 raft_consensus.cc:979] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:13.655262 12568 raft_consensus.cc:1067] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:13.673249 10340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a123878203f45319def3c4246b9fe30"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:13.674088 10340 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:13.674594 10340 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.680541 12588 raft_consensus.cc:979] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:13.681529 12588 raft_consensus.cc:1067] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:13.685446 12586 raft_consensus.cc:979] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:13.687793 12445 raft_consensus.cc:1067] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:13.628814 12563 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20240507 08:06:13.689720 12560 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:13.726114 10335 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:13.726776 10335 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:13.727176 10335 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.729949 10339 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:13.730542 10339 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:13.730931 10339 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.757278 12563 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20240507 08:06:13.755549 10339 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.756569 10340 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.766985 10335 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.770576 10335 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:13.770728 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:13.771524 10191 raft_consensus.cc:3032] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:13.771842 10191 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:13.772706 10191 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.776861 10191 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.790016 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:13.792038 10183 raft_consensus.cc:3032] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:13.792435 10183 raft_consensus.cc:729] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:13.793704 10183 consensus_queue.cc:260] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.796093 10183 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.796909 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:13.797613 10184 raft_consensus.cc:3032] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:13.797925 10184 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:13.798785 10184 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.800248 10184 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.803613 10340 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:13.806169 10339 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:13.818830 10191 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:13.824488 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:13.825204 10265 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.826242 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:13.826803 10266 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.825068 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:13.829689 10261 raft_consensus.cc:3037] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:13.830543 10184 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:13.834328 10306 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:13.838889 10306 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:13.839406 12560 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:13.840031 12560 raft_consensus.cc:686] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:13.840376 12554 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:13.840912 12554 raft_consensus.cc:686] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:13.839958 12565 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:13.841903 12554 consensus_queue.cc:237] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.841296 12560 consensus_queue.cc:237] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.855861 10265 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:13.856588 10191 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:13.867100 10337 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:13.815371 10090 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:13.874994 12567 raft_consensus.cc:1050] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:06:13.878105 10261 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:13.877544 10266 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:13.882951 12567 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:13.883383 10304 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:13.876096 12060 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20240507 08:06:13.886430 12563 raft_consensus.cc:2781] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:13.890719 12563 raft_consensus.cc:686] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:13.891800 12563 consensus_queue.cc:237] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:13.898237 10185 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:13.904089 12480 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:13.907050 10940 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:13.914737 10260 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:13.959055 10183 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:13.978950 10939 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:13.980886 12567 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:13.993269 10188 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:13.998095 10266 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:13.998909 10337 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:14.003746 10188 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:14.004585 12522 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:14.005158 12480 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:14.007107 10337 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:14.010390 10183 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:14.011929 12526 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:14.022056 12526 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:14.026005 12522 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:14.046308 10943 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Advancing to term 7
I20240507 08:06:14.057485 10943 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:14.101820 12522 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.016s
I20240507 08:06:14.142544 12565 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.151466 10188 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:14.153924 10266 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.170928 12569 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20240507 08:06:14.182230 10089 catalog_manager.cc:5484] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:14.201539 10086 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.209548 10089 catalog_manager.cc:5484] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:14.148155 10943 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.297938 10088 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
W20240507 08:06:14.303651 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 87118.2us
W20240507 08:06:14.360889 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 96519.4us
I20240507 08:06:14.384132 10265 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:14.384579 12620 raft_consensus.cc:1050] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:06:14.385413 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 75172.9us
I20240507 08:06:14.397636 12563 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:14.392670 12620 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:14.449105 10184 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:14.448748 10191 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.464888 10266 raft_consensus.cc:1261] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:14.467116 10264 raft_consensus.cc:1261] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:14.471053 12563 consensus_queue.cc:1035] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:14.472362 12563 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:14.473958 10185 raft_consensus.cc:1261] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:14.506311 12526 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:14.510965 10089 catalog_manager.cc:5484] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.497815 12563 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20240507 08:06:14.539366 10940 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:14.541224 10332 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:14.541409 12522 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:14.542481 12620 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:14.547053 10191 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:14.548214 12522 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:14.556155 12622 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.566653 10086 catalog_manager.cc:5484] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:14.569607 10185 raft_consensus.cc:1261] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:14.571144 10086 catalog_manager.cc:5484] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:14.582597 10188 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.588101 10337 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.592659 10940 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 4 to 7, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.644721 10340 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:14.651047 10089 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 4 to 7, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.664181 12526 consensus_queue.cc:1035] T 7a123878203f45319def3c4246b9fe30 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:14.665614 10332 consensus_queue.cc:237] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:14.667384 10940 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:14.668951 12526 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:14.677965 10943 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:14.682471 12450 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:14.686707 12526 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.693271 12560 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.694486 10940 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.699381 10943 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:06:14.734175 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 64116.3us
I20240507 08:06:14.752565 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85b6fa5360054f32af996ec68d175584 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:14.790035 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e940ba6acb8d4782a4c921b50b882f91 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:14.796583 10260 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:14.815528 12526 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:14.822803 10191 raft_consensus.cc:1261] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:14.825173 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:14.832008 10260 raft_consensus.cc:2932] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.841560 12526 consensus_queue.cc:1035] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:14.855474 10090 catalog_manager.cc:5484] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.858763 10185 raft_consensus.cc:2932] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.876677 10087 catalog_manager.cc:5484] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:14.891633 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet e940ba6acb8d4782a4c921b50b882f91 with delete_type TABLET_DATA_TOMBSTONED (TS 59382e6647c549d89d83c70233c0667e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55810
I20240507 08:06:14.911957 12637 tablet_replica.cc:330] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: stopping tablet replica
I20240507 08:06:14.968434 12637 raft_consensus.cc:2229] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:14.969298 12637 raft_consensus.cc:2258] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:14.970450 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet 85b6fa5360054f32af996ec68d175584 with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42910
I20240507 08:06:14.977036 12637 ts_tablet_manager.cc:1875] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:14.991642 12638 tablet_replica.cc:330] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:06:14.996440 12637 ts_tablet_manager.cc:1888] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20240507 08:06:14.997140 12637 log.cc:1196] T e940ba6acb8d4782a4c921b50b882f91 P 59382e6647c549d89d83c70233c0667e: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/wals/e940ba6acb8d4782a4c921b50b882f91
I20240507 08:06:15.002561 12638 raft_consensus.cc:2229] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:14.999253 10076 catalog_manager.cc:4830] TS 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): tablet e940ba6acb8d4782a4c921b50b882f91 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:15.008569 12638 raft_consensus.cc:2258] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:15.021538 12638 ts_tablet_manager.cc:1875] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:15.040362 10263 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:15.056488 10943 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:15.057690 12060 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:15.063625 10184 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:15.065346 12060 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:15.080989 12638 ts_tablet_manager.cc:1888] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20240507 08:06:15.081012 12622 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:15.087612 12638 log.cc:1196] T 85b6fa5360054f32af996ec68d175584 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/85b6fa5360054f32af996ec68d175584
I20240507 08:06:15.083479 10943 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:15.089538 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet 85b6fa5360054f32af996ec68d175584 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:15.085917 10184 raft_consensus.cc:2932] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 7 to 8, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:15.132406 10183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:15.133205 10183 raft_consensus.cc:595] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:15.150368 10087 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 7 to 8, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:15.151847 10260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:15.152535 10260 raft_consensus.cc:595] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:15.157407 10340 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:15.158001 10340 raft_consensus.cc:595] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:15.163441 10260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:15.164126 10260 raft_consensus.cc:595] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:15.165490 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240507 08:06:15.170158 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:15.170892 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:15.172785 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet 9800706282e543be9f2ee3473da2b89f with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:15.194207 12565 raft_consensus.cc:979] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:15.194718 12565 raft_consensus.cc:1067] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:15.196820 10184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55820
I20240507 08:06:15.197613 10184 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:15.198104 10184 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.207152 10184 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.220149 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:15.220878 10264 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.231515 10184 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:15.229307 10264 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:15.236317 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:15.234391 10155 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:15.236897 10337 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:15.237205 10337 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:15.238188 10337 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.240370 10337 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.244851 12631 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:15.246006 12631 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:15.247133 12631 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.261004 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9800706282e543be9f2ee3473da2b89f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:15.267983 10337 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:15.295212 12639 tablet_replica.cc:330] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:06:15.302306 12060 raft_consensus.cc:979] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:15.303050 12312 raft_consensus.cc:1067] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:15.304682 10332 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39710
I20240507 08:06:15.305200 10332 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:15.305510 10332 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:15.315430 10332 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.321550 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:15.322276 10183 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:15.322963 10332 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:15.327190 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:15.327729 10264 raft_consensus.cc:3032] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:15.327956 10264 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:15.331199 10264 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.336494 10264 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:15.351323 10264 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:15.353914 10304 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:15.359648 12565 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:15.360064 12565 raft_consensus.cc:686] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:15.360698 12565 consensus_queue.cc:237] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.362610 10088 catalog_manager.cc:5484] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:15.369928 10183 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:15.370507 12625 raft_consensus.cc:979] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:15.372051 12445 raft_consensus.cc:1067] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:15.374464 12639 raft_consensus.cc:2229] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:15.382404 12639 raft_consensus.cc:2258] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:15.386685 12639 ts_tablet_manager.cc:1875] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:15.375947 10340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:15.390110 10340 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:15.390488 10340 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.395210 10088 catalog_manager.cc:5484] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 7 to 8, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 8 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:15.414273 12569 raft_consensus.cc:979] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:15.414817 12569 raft_consensus.cc:1067] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:15.402704 10340 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.417443 10183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55816
I20240507 08:06:15.417999 10183 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:15.418299 10183 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.424829 10183 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.430575 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:15.447422 10184 raft_consensus.cc:3032] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:15.447810 10184 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:15.448602 10184 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.449834 10184 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.431404 10340 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:15.432327 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:15.439702 12639 ts_tablet_manager.cc:1888] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.7
I20240507 08:06:15.455945 10264 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.455972 12639 log.cc:1196] T 9800706282e543be9f2ee3473da2b89f P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/9800706282e543be9f2ee3473da2b89f
I20240507 08:06:15.459793 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet 9800706282e543be9f2ee3473da2b89f (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:15.468798 10264 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:15.473695 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:15.473950 10184 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:15.475489 12565 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:15.476001 12565 raft_consensus.cc:686] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:15.477032 10183 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:15.476873 12565 consensus_queue.cc:237] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.482575 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:15.483150 10260 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:15.483458 10260 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:15.484207 10260 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.485641 10260 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.488968 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:15.489506 10340 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:15.492842 10260 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:15.496899 10340 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:15.497346 10155 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:15.498760 12631 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:15.499218 12631 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:15.499984 12631 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:15.501989 10088 catalog_manager.cc:5484] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 6 to 7, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:15.529750 10088 catalog_manager.cc:5484] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 7, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:15.675158 10260 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:15.677350 12631 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:15.697360 10340 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:15.698803 12631 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:15.766853 10184 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:15.768852 12565 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:15.835440 10260 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:15.838048 12465 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:15.919312 10263 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:15.920611 12465 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:15.923400 10184 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:15.926581 12465 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:15.983633 10263 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:15.985000 12625 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:16.006242 10332 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:16.010299 12631 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:16.113209 10263 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
W20240507 08:06:16.147837 10233 consensus_peers.cc:485] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d3f101fe713f47b1b84bcaeb54f7aead. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:16.150187 10332 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:16.151952 10184 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:16.153486 12060 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:16.158838 12060 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:16.199653 12060 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.206053 10184 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.207180 10337 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.224510 10089 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index -1 to 4, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 4 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.238384 10183 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:16.272497 10337 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20240507 08:06:16.273020 10158 consensus_peers.cc:485] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db7c60767d74458094c1d86a49e87980. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:16.274592 10263 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:16.275422 12631 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:16.276398 12586 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:16.394538 12586 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.396504 10260 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.413501 10337 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.434816 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:16.440126 10263 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:16.485394 10089 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:16.505051 10340 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:16.507172 10185 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:16.507737 12312 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:16.509617 12060 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240507 08:06:16.510403 10233 consensus_peers.cc:485] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a181f4ae4d342129ad262e5545debc9. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:16.528462 12659 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.529865 10340 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.534566 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:16.536650 10185 raft_consensus.cc:595] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:16.541734 10332 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:16.542276 10332 raft_consensus.cc:595] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:16.545125 10263 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:16.545761 10263 raft_consensus.cc:595] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:16.550887 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240507 08:06:16.552946 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:16.553702 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:16.563973 10087 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.582682 10185 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:16.629340 10184 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.654633 10337 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:16.684962 12631 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240507 08:06:16.709439 10158 consensus_peers.cc:485] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03d2e621edea41ae97e79e36e4dc0365. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:16.728089 12672 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.734205 10340 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.762140 12312 raft_consensus.cc:979] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:16.762743 12312 raft_consensus.cc:1067] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:16.764231 10337 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39710
I20240507 08:06:16.764791 10337 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:16.765136 10337 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:16.771764 10337 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:16.773854 10337 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:16.774783 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:16.775403 10260 raft_consensus.cc:3032] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:16.775720 10260 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:16.776460 10260 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:16.778328 12675 ts_tablet_manager.cc:902] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:16.778584 10260 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:16.781334 10263 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:16.787031 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:16.787571 10185 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:16.792474 10260 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:16.796343 10304 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:16.795537 10185 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:16.800017 12670 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:16.800495 12670 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:16.801061 10089 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.801311 12670 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:16.807495 12672 raft_consensus.cc:979] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:16.807987 12671 raft_consensus.cc:1067] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:16.809051 12675 tablet_copy_client.cc:316] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:06:16.810843 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet d3f101fe713f47b1b84bcaeb54f7aead from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:06:16.811383 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet d3f101fe713f47b1b84bcaeb54f7aead from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-d3f101fe713f47b1b84bcaeb54f7aead
I20240507 08:06:16.814060 12674 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:16.820816 10264 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:16.821731 10264 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:16.822062 10264 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:16.823916 10087 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:16.829346 10264 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:16.831789 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:16.832394 10184 raft_consensus.cc:3032] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:16.832693 10184 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:16.833382 10184 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:16.836061 10264 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:16.837842 10184 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:16.842857 10277 tablet_copy_source_session.cc:205] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:16.843415 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:16.848965 10184 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:16.852684 10340 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:16.854142 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:16.854996 12659 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:16.855466 12659 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:16.856300 12659 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:16.864919 10340 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:16.879241 10260 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index -1 to 5, NON_VOTER 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } } }
I20240507 08:06:16.892416 12676 ts_tablet_manager.cc:902] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:06:16.904111 12675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f101fe713f47b1b84bcaeb54f7aead. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:06:16.920917 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 51925.3us
I20240507 08:06:16.936602 12670 raft_consensus.cc:979] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:16.937103 12565 raft_consensus.cc:1067] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:16.939718 10087 catalog_manager.cc:5484] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:16.942198 10185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55820
I20240507 08:06:16.942731 10185 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:16.943056 10185 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:16.969406 12676 tablet_copy_client.cc:316] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:06:16.968711 10185 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:16.978871 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:16.979494 10260 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:16.990022 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:16.990571 10337 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:16.990828 10337 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:16.991048 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet db7c60767d74458094c1d86a49e87980 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:06:16.991556 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet db7c60767d74458094c1d86a49e87980 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-db7c60767d74458094c1d86a49e87980
I20240507 08:06:16.991489 10337 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:16.992691 10337 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:17.007511 10202 tablet_copy_source_session.cc:205] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:17.011866 12676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7c60767d74458094c1d86a49e87980. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:17.014418 10185 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:17.015383 10260 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:17.022413 10155 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:17.023169 12674 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:17.023620 12674 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:17.019034 10337 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:17.024456 12674 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:17.034682 12675 tablet_copy_client.cc:791] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:17.065943 12675 tablet_copy_client.cc:655] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:17.071712 12675 tablet_copy_client.cc:531] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:17.092103 10087 catalog_manager.cc:5484] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 7 to 8, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 8 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:17.093434 12675 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:17.110749 12676 tablet_copy_client.cc:791] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:17.111557 12676 tablet_copy_client.cc:655] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:17.121233 12676 tablet_copy_client.cc:531] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:17.147990 12686 ts_tablet_manager.cc:902] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:17.168546 12676 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:17.195582 12686 tablet_copy_client.cc:316] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:06:17.203038 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet 7a181f4ae4d342129ad262e5545debc9 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:06:17.203606 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet 7a181f4ae4d342129ad262e5545debc9 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-7a181f4ae4d342129ad262e5545debc9
I20240507 08:06:17.224004 10184 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:17.225656 12681 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:17.235072 10277 tablet_copy_source_session.cc:205] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:17.238296 12675 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:17.242029 12686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a181f4ae4d342129ad262e5545debc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:17.245230 12675 tablet_bootstrap.cc:492] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:17.246114 12675 ts_tablet_manager.cc:1367] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.153s user 0.024s sys 0.015s
I20240507 08:06:17.258405 10337 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:17.249234 12675 raft_consensus.cc:348] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:17.259470 12659 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:17.259945 12675 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:17.268924 12690 ts_tablet_manager.cc:902] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795)
I20240507 08:06:17.278712 10265 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:17.260602 12675 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:17.285185 12565 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240507 08:06:17.313117 12675 ts_tablet_manager.cc:1398] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.067s user 0.006s sys 0.001s
I20240507 08:06:17.317090 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-d3f101fe713f47b1b84bcaeb54f7aead received from {username='slave'} at 127.0.0.1:39118
I20240507 08:06:17.317500 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-d3f101fe713f47b1b84bcaeb54f7aead on tablet d3f101fe713f47b1b84bcaeb54f7aead with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:17.339344 12676 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:17.360476 12676 tablet_bootstrap.cc:492] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:17.396266 10940 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:17.398609 12676 ts_tablet_manager.cc:1367] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.230s user 0.021s sys 0.014s
I20240507 08:06:17.374226 12690 tablet_copy_client.cc:316] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.129:32795
I20240507 08:06:17.413622 12465 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240507 08:06:17.420953 12686 tablet_copy_client.cc:791] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:17.421643 12686 tablet_copy_client.cc:655] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:17.420969 12676 raft_consensus.cc:348] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:17.422144 12676 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:17.422835 12676 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:17.424712 12676 ts_tablet_manager.cc:1398] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20240507 08:06:17.427405 10183 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:17.432191 10202 tablet_copy_service.cc:140] P 59382e6647c549d89d83c70233c0667e: Received BeginTabletCopySession request for tablet 03d2e621edea41ae97e79e36e4dc0365 from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:50108)
I20240507 08:06:17.451576 10202 tablet_copy_service.cc:161] P 59382e6647c549d89d83c70233c0667e: Beginning new tablet copy session on tablet 03d2e621edea41ae97e79e36e4dc0365 from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:50108: session id = 5d6656a616b747d393ff3de5328d2540-03d2e621edea41ae97e79e36e4dc0365
I20240507 08:06:17.457856 10201 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-db7c60767d74458094c1d86a49e87980 received from {username='slave'} at 127.0.0.1:50108
I20240507 08:06:17.458267 10201 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-db7c60767d74458094c1d86a49e87980 on tablet db7c60767d74458094c1d86a49e87980 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:17.477497 10202 tablet_copy_source_session.cc:205] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:17.484608 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03d2e621edea41ae97e79e36e4dc0365. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:06:17.513934 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 125434us
I20240507 08:06:17.523723 10260 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:17.525434 12672 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:17.534693 12686 tablet_copy_client.cc:531] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:17.511445 12700 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20240507 08:06:17.589275 12686 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:17.602015 10340 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:17.608129 12586 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:17.653877 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:17.667702 10943 raft_consensus.cc:1203] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.4->[7.5-7.5] Dedup: 7.5->[]
I20240507 08:06:17.719384 12702 raft_consensus.cc:1050] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:06:17.704466 12706 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:06:17.729818 12702 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:17.745044 10184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:17.745749 10184 raft_consensus.cc:595] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:17.760129 10260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:17.760754 10260 raft_consensus.cc:595] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:17.763643 10940 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:17.763643 10263 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:17.765465 10332 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:17.766064 10332 raft_consensus.cc:595] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:17.771562 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240507 08:06:17.773242 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:17.773929 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:17.773936 12702 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:17.774776 12679 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:17.768707 12690 tablet_copy_client.cc:791] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:17.785176 10183 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:17.786581 12680 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:17.782645 12690 tablet_copy_client.cc:655] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:17.804663 12690 tablet_copy_client.cc:531] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:17.804853 12680 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:17.805879 10183 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:17.819707 12672 raft_consensus.cc:979] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:17.820235 12672 raft_consensus.cc:1067] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:17.826535 10263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:17.827147 10263 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:17.826211 10940 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:17.827479 10263 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:17.844621 10088 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e reported cstate change: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:17.867060 10260 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 4 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:17.867398 10263 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:17.870980 10263 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:17.871322 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:17.871589 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:17.871924 10340 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:17.872150 10185 raft_consensus.cc:3032] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:17.872457 10185 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:17.874294 10185 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:17.875435 10185 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:17.877199 12686 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:17.885808 12686 tablet_bootstrap.cc:492] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:17.889559 12690 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:17.892257 10340 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:17.893821 10231 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:17.894776 10185 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:17.903091 12704 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:17.903616 12704 raft_consensus.cc:686] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:17.904400 12704 consensus_queue.cc:237] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:17.901226 12686 ts_tablet_manager.cc:1367] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.309s user 0.032s sys 0.032s
I20240507 08:06:17.920379 12686 raft_consensus.cc:348] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:17.921248 12686 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:17.949797 12686 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:17.959331 12686 ts_tablet_manager.cc:1398] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.041s user 0.005s sys 0.000s
I20240507 08:06:17.961802 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-7a181f4ae4d342129ad262e5545debc9 received from {username='slave'} at 127.0.0.1:39118
I20240507 08:06:17.962215 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-7a181f4ae4d342129ad262e5545debc9 on tablet 7a181f4ae4d342129ad262e5545debc9 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:18.061733 12465 raft_consensus.cc:979] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:18.062285 12679 raft_consensus.cc:1067] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:18.071003 10183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55820
I20240507 08:06:18.071738 10183 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:18.072198 10183 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:18.088208 10332 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:18.101632 10940 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:18.102952 10260 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:18.104605 12465 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:18.105929 12450 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:18.111661 10089 catalog_manager.cc:5484] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 3, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 3 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:18.115492 12450 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.118685 10940 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.120347 10260 raft_consensus.cc:2932] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.144356 12586 raft_consensus.cc:1050] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:06:18.151298 12586 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 3, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:18.157478 10943 raft_consensus.cc:1203] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20240507 08:06:18.195286 10075 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d3f101fe713f47b1b84bcaeb54f7aead with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:18.200320 10183 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:18.228456 10086 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 6 to 7, VOTER 59382e6647c549d89d83c70233c0667e (127.8.213.129) evicted. New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.230055 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:18.247221 10339 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:18.248188 10260 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:18.274418 10940 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:18.295886 12586 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:18.298529 12674 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:18.324934 12703 raft_consensus.cc:979] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:18.325452 12703 raft_consensus.cc:1067] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:18.327549 10340 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39710
I20240507 08:06:18.328194 10340 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:18.330247 10340 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:18.281385 10183 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:18.290011 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:18.341342 10943 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:18.294363 10332 raft_consensus.cc:3032] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:18.368755 10332 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:18.369755 10332 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:18.370995 10332 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:18.373330 10943 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:18.427999 10340 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:18.428975 10158 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:18.429843 10182 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f101fe713f47b1b84bcaeb54f7aead with delete_type TABLET_DATA_TOMBSTONED (TS 59382e6647c549d89d83c70233c0667e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:55810
I20240507 08:06:18.431260 10332 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:18.432976 12711 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240507 08:06:18.435926 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:18.436523 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:18.436786 10188 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:18.437104 10264 raft_consensus.cc:3032] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:18.437757 10264 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:18.438302 10264 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:18.439196 10264 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:18.442337 12690 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:06:18.448611 12690 tablet_bootstrap.cc:492] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:18.450438 12690 ts_tablet_manager.cc:1367] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 0.561s user 0.029s sys 0.004s
I20240507 08:06:18.450579 10337 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:18.452642 12704 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:18.454535 12586 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.447568 10264 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:18.461676 10340 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:18.462987 12674 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:18.463474 12674 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:18.464841 12690 raft_consensus.cc:348] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:18.465485 10304 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:18.472977 12681 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:18.476002 12681 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:18.476926 12681 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:18.477226 12674 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:18.466934 10339 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.474380 10183 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:18.473446 10188 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:18.507988 12720 tablet_replica.cc:330] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: stopping tablet replica
I20240507 08:06:18.484826 12690 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:18.497996 12700 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20240507 08:06:18.538555 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 49410.7us
I20240507 08:06:18.577739 12720 raft_consensus.cc:2229] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:18.586807 12720 raft_consensus.cc:2258] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:18.595582 12720 ts_tablet_manager.cc:1875] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:18.549469 12690 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
W20240507 08:06:18.643014 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 82632.2us
W20240507 08:06:18.665877 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 57492.8us
I20240507 08:06:18.663547 10086 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 6, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240507 08:06:18.669384 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 118504us
I20240507 08:06:18.669607 10086 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.674708 10260 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
W20240507 08:06:18.679951 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 67540.7us
I20240507 08:06:18.686537 12690 ts_tablet_manager.cc:1398] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.236s user 0.006s sys 0.003s
I20240507 08:06:18.690217 10202 tablet_copy_service.cc:342] P 59382e6647c549d89d83c70233c0667e: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-03d2e621edea41ae97e79e36e4dc0365 received from {username='slave'} at 127.0.0.1:50108
I20240507 08:06:18.690636 10202 tablet_copy_service.cc:433] P 59382e6647c549d89d83c70233c0667e: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-03d2e621edea41ae97e79e36e4dc0365 on tablet 03d2e621edea41ae97e79e36e4dc0365 with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:18.694634 10088 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:18.695683 10943 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.748929 10943 raft_consensus.cc:1203] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.4->[5.5-5.5] Dedup: 5.5->[]
I20240507 08:06:18.788265 12720 ts_tablet_manager.cc:1888] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20240507 08:06:18.788743 12720 log.cc:1196] T d3f101fe713f47b1b84bcaeb54f7aead P 59382e6647c549d89d83c70233c0667e: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-0-root/wals/d3f101fe713f47b1b84bcaeb54f7aead
I20240507 08:06:18.797792 10076 catalog_manager.cc:4830] TS 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): tablet d3f101fe713f47b1b84bcaeb54f7aead (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:18.813774 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:18.814356 10339 raft_consensus.cc:595] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership
I20240507 08:06:18.839063 10185 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:18.854722 10260 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:18.858830 10943 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:18.859282 10340 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:18.859309 12586 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:18.859866 10340 raft_consensus.cc:595] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership
W20240507 08:06:18.861357 10075 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 7a181f4ae4d342129ad262e5545debc9 already in progress [suppressed 1 similar messages]
I20240507 08:06:18.866292 12731 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:18.874941 12586 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.880560 10943 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.881419 10264 raft_consensus.cc:2932] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 6 to 7, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.892314 10260 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240507 08:06:18.901793 12679 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:18.902616 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:18.902825 10940 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:18.900638 10088 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 6 to 7, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:18.904417 12674 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240507 08:06:18.907018 10075 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45703.4us
I20240507 08:06:18.908412 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:18.908931 10339 raft_consensus.cc:595] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Received request to transfer leadership
W20240507 08:06:18.911454 10138 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 7a181f4ae4d342129ad262e5545debc9 already in progress
W20240507 08:06:18.911767 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7a181f4ae4d342129ad262e5545debc9 already in progress
I20240507 08:06:18.933434 10076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db7c60767d74458094c1d86a49e87980 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:18.942294 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet db7c60767d74458094c1d86a49e87980 with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:19.001287 12743 tablet_replica.cc:330] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:06:19.002355 12743 raft_consensus.cc:2229] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:19.003103 12743 raft_consensus.cc:2258] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:19.016629 10185 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240507 08:06:19.018016 12679 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240507 08:06:19.018509 12742 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20240507 08:06:19.019655 10260 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:19.020203 10260 raft_consensus.cc:595] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:19.027241 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:19.029837 10185 raft_consensus.cc:595] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:19.048501 12727 raft_consensus.cc:979] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:19.049490 12727 raft_consensus.cc:1067] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:19.054999 10940 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Advancing to term 6
I20240507 08:06:19.056195 10264 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42924
I20240507 08:06:19.056891 10264 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:19.057233 10264 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:19.061692 10340 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:19.063375 12743 ts_tablet_manager.cc:1875] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:19.063649 12586 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:19.072520 10940 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 6 LEARNER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240507 08:06:19.073006 10264 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:19.076000 10260 raft_consensus.cc:1226] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:19.091184 12465 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:19.107077 10264 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.275807 12748 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.276548 12748 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.279101 12748 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:20.392267 9051 debug-util.cc:398] Leaking SignalData structure 0x7b080000ed20 after lost signal to thread 10188
W20240507 08:06:20.392895 9051 kernel_stack_watchdog.cc:198] Thread 10188 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 355ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20240507 08:06:20.591665 12748 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.592648 12748 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.741387 12748 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.741961 12748 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.742986 12750 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.745229 12749 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.745726 12749 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.745339 12750 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.747397 12749 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.747910 12749 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.748106 12750 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.748283 12749 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.748857 12750 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.743914 12748 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.749894 12749 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.750571 12749 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.750839 12586 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.751190 12749 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.751451 12586 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.752120 12730 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.752997 12586 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.752617 12730 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.753307 12749 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.753906 12749 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.754321 12749 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.755076 12730 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.755554 12730 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.756186 12749 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.755934 12730 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.757567 12730 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.753379 12586 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.759737 12586 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.749447 12750 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.760994 12586 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.759301 12748 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.761482 12748 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:19.112250 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:20.761543 12750 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.762228 10339 raft_consensus.cc:3032] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:20.762308 12750 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.762816 10339 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:20.762176 12748 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.762948 12750 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.763572 10339 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:20.764400 12748 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.764642 10339 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:20.764953 12748 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.765425 12750 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.765425 12748 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:20.766332 12750 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.766813 12750 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.767439 12748 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.767890 12748 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.768484 12750 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.768275 12748 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.770072 12748 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.770947 10339 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:20.771371 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1452) took 1659 ms. Trace:
I20240507 08:06:20.771755 10339 rpcz_store.cc:276] 0507 08:06:19.111703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.111947 (+ 244us) service_pool.cc:225] Handling call
0507 08:06:20.771282 (+1659335us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:20.784507 12748 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:20.785022 12748 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.786875 12748 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.787470 12748 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.787914 12748 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.789795 12748 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.795241 12751 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.795351 12750 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.795451 12748 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:20.795786 12751 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.796432 12748 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.797194 12679 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.797724 12679 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.798223 12743 ts_tablet_manager.cc:1888] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20240507 08:06:20.798332 12751 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.796131 12750 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.798691 12743 log.cc:1196] T db7c60767d74458094c1d86a49e87980 P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/db7c60767d74458094c1d86a49e87980
I20240507 08:06:20.798960 12748 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.799599 12679 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.799708 12743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39708 (request call id 353) took 1857 ms. Trace:
I20240507 08:06:20.800094 12750 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.800506 12679 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.800145 12743 rpcz_store.cc:276] 0507 08:06:18.941983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:18.942122 (+ 139us) service_pool.cc:225] Handling call
0507 08:06:19.000974 (+ 58852us) ts_tablet_manager.cc:1120] Deleting tablet db7c60767d74458094c1d86a49e87980
0507 08:06:19.001031 (+ 57us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:06:19.202296 (+201265us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:20.797952 (+1595656us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:20.799641 (+ 1689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":58167,"thread_start_us":65876,"threads_started":1}
I20240507 08:06:20.801141 12679 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.803205 12679 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.803943 12679 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.804404 12679 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.806221 12679 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:20.808969 10939 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.663s user 0.013s sys 0.027s
W20240507 08:06:20.809465 10939 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.664s user 0.014s sys 0.028s
I20240507 08:06:20.816846 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.818286 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.818981 10190 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:20.819144 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.818375 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.822190 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.824934 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.825655 10186 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
W20240507 08:06:20.828438 10121 debug-util.cc:398] Leaking SignalData structure 0x7b0800312b80 after lost signal to thread 9049
I20240507 08:06:20.830080 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.831099 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.832592 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.833235 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.835795 10186 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:20.838227 12755 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.847137 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.847890 10186 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:20.847720 12755 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.848908 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.845019 12558 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.849835 12558 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.850579 12729 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:20.851187 12729 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.851877 12558 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.852430 12758 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.854473 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.854771 12758 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.847970 12731 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:20.840976 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.852478 10187 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:20.845882 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.853829 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.857599 10189 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:20.846767 12752 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.858115 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:20.856647 10185 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:20.857106 10186 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:20.859225 12733 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.859436 12672 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:20.833138 12754 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.858520 12752 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.864261 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.866323 12672 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.870580 12672 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.872565 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:20.875226 12752 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.867216 12754 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.867157 12725 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:20.877084 12725 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.871398 12733 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.878095 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.879580 12733 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.879962 12754 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:19.128870 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:20.877485 12725 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:20.881111 12729 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:20.881278 10191 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1522) took 1752 ms (client timeout 2227 ms). Trace:
W20240507 08:06:20.881825 10191 rpcz_store.cc:269] 0507 08:06:19.128429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.128586 (+ 157us) service_pool.cc:225] Handling call
0507 08:06:20.881182 (+1752596us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:20.851150 12755 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.882843 12679 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:20.882983 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:20.883339 12679 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.883781 10306 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:20.855739 12731 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.886425 12702 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:20.886793 12702 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.887123 12702 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:20.888270 12731 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.919462 10939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 257) took 1802 ms. Trace:
I20240507 08:06:20.919906 10939 rpcz_store.cc:276] 0507 08:06:19.116696 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.126265 (+ 9569us) service_pool.cc:225] Handling call
0507 08:06:19.129960 (+ 3695us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:19.136924 (+ 6964us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:19.136971 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:19.137524 (+ 553us) log.cc:842] Serialized 26 byte log entry
0507 08:06:20.809878 (+1672354us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:20.844119 (+ 34241us) log.cc:842] Serialized 14 byte log entry
0507 08:06:20.918579 (+ 74460us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:20.919151 (+ 572us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:20.919209 (+ 58us) raft_consensus.cc:1671] finished
0507 08:06:20.919221 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:20.919380 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34168,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":233,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":341,"thread_start_us":1771439,"threads_started":3,"wal-append.queue_time_us":150645,"wal-append.run_cpu_time_us":1488,"wal-append.run_wall_time_us":1001497}
I20240507 08:06:20.883850 12679 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:20.925810 12761 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.926326 12761 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:20.930511 10188 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.895s user 0.016s sys 0.031s
W20240507 08:06:20.931255 10188 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.895s user 0.017s sys 0.031s
I20240507 08:06:20.931980 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1871) took 1912 ms. Trace:
I20240507 08:06:20.932278 10188 rpcz_store.cc:276] 0507 08:06:19.019933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.031238 (+ 11305us) service_pool.cc:225] Handling call
0507 08:06:19.031534 (+ 296us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:19.031876 (+ 342us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:19.031903 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:19.035786 (+ 3883us) log.cc:842] Serialized 26 byte log entry
0507 08:06:20.931624 (+1895838us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:20.931666 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:20.931746 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:20.931788 (+ 42us) raft_consensus.cc:1671] finished
0507 08:06:20.931799 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:20.931904 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3708,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9048,"thread_start_us":1895579,"threads_started":1,"wal-append.queue_time_us":171036,"wal-append.run_cpu_time_us":1592,"wal-append.run_wall_time_us":1003562}
W20240507 08:06:20.935869 10943 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.868s user 0.024s sys 0.031s
W20240507 08:06:20.936424 10943 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.869s user 0.024s sys 0.031s
I20240507 08:06:20.937847 12749 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.938381 12749 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:20.939041 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 265) took 1016 ms. Trace:
I20240507 08:06:20.939397 10183 rpcz_store.cc:276] 0507 08:06:19.922203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:20.938323 (+1016120us) service_pool.cc:225] Handling call
0507 08:06:20.938602 (+ 279us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:20.938760 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:20.938784 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:20.938821 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:20.938849 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:20.938897 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:20.938992 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:20.942099 12060 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:20.942600 12060 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.943002 12736 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.943466 12736 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.945124 12704 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.945240 12736 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.945704 12704 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.948223 12060 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.949196 12748 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.962402 12704 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.962988 12704 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.963461 12704 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.960348 12748 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.971465 12748 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.972066 12748 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.972569 12748 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:20.957878 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 71663.8us
I20240507 08:06:20.979499 12465 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.948426 12727 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:20.949965 12755 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.950688 12751 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:20.991477 12751 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.992991 12748 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:20.992991 12731 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:20.993567 12755 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.993769 12731 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:20.995532 12731 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:20.965425 12704 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:20.979995 12465 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.989804 12727 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:20.990885 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 1876) took 1068 ms. Trace:
I20240507 08:06:20.953433 12702 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.996937 12558 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.006423 12558 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.001564 10192 rpcz_store.cc:276] 0507 08:06:19.922774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:20.989807 (+1067033us) service_pool.cc:225] Handling call
0507 08:06:20.990172 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:20.990355 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:20.990392 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:20.990442 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:20.990478 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:20.990548 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:20.990709 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:20.997537 10943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 146) took 1930 ms. Trace:
I20240507 08:06:21.005424 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1689) took 1943 ms. Trace:
I20240507 08:06:21.009042 12465 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.009630 12465 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.009191 10260 rpcz_store.cc:276] 0507 08:06:19.055620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.072043 (+ 16423us) service_pool.cc:225] Handling call
0507 08:06:19.072542 (+ 499us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:19.090248 (+ 17706us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:20.999226 (+1908978us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b54002909d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:20.999428 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7591168}
I20240507 08:06:21.009083 10943 rpcz_store.cc:276] 0507 08:06:19.066735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.066854 (+ 119us) service_pool.cc:225] Handling call
0507 08:06:19.067135 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:19.067326 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 8
0507 08:06:20.936750 (+1869424us) log.cc:842] Serialized 14 byte log entry
0507 08:06:20.996145 (+ 59395us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:20.996226 (+ 81us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:20.996802 (+ 576us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:20.997335 (+ 533us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:20.997475 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":148879,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":77,"raft.run_wall_time_us":3593,"spinlock_wait_cycles":7328640,"thread_start_us":1921487,"threads_started":2,"wal-append.queue_time_us":52698}
I20240507 08:06:21.016615 12465 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:20.995532 12755 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.024550 12764 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.025065 12764 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.025549 12704 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.026072 12704 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.026736 12736 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.027160 12764 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.027982 12704 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.027451 12736 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.028640 12703 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.029142 12703 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:21.030614 12060 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.031226 12703 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.031322 12060 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.031821 12703 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.032303 12703 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.033157 12060 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.033720 12060 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.034116 12060 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.034644 12727 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.030679 12704 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.004460 12730 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.041694 12060 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.012966 12731 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.016949 12751 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.045037 12465 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.048827 12728 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.048969 12725 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.050948 12725 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.051450 12731 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:21.053557 12731 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.054044 12731 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.054312 12758 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.054505 12731 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.055743 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.057250 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.047794 12730 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.049333 12728 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.089897 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.089993 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.090488 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.090610 10339 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:21.091099 10332 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:21.091464 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.091754 12760 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.091969 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.092285 12760 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.092913 12730 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.083125 12729 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:21.084380 10121 debug-util.cc:398] Leaking SignalData structure 0x7b08000acb60 after lost signal to thread 10124
I20240507 08:06:21.093967 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.091532 12728 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.089463 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.096378 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.097713 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.098245 10341 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:21.098572 12703 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.098867 12700 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.099089 12703 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.099610 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.100337 12702 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.088965 10087 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59954 (request call id 207) took 2013 ms. Trace:
I20240507 08:06:21.099607 12703 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.004982 12679 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.100965 10087 rpcz_store.cc:276] 0507 08:06:19.075205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:19.075365 (+ 160us) service_pool.cc:225] Handling call
0507 08:06:21.088888 (+2013523us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":748,"reported_tablets":1}
I20240507 08:06:21.101820 12700 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.103734 12700 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.104187 12700 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.104625 12700 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.105311 12679 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.108246 12749 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.109638 12703 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.110157 12764 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.110622 12764 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.099136 12760 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.110899 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.112023 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.112529 12764 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.086751 12761 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.113838 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.086527 12731 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.086553 12736 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.086908 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.114413 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.114939 10340 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:06:21.087633 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.047960 12704 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.115731 10939 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:21.114522 10341 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:21.116871 12704 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.093219 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.117295 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.117893 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.118039 12558 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.119123 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.119374 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.120731 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.121655 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.093967 12729 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.122241 10341 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:21.127341 10307 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:21.129781 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.134469 12674 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:21.135347 10121 debug-util.cc:398] Leaking SignalData structure 0x7b0800064da0 after lost signal to thread 10216
I20240507 08:06:21.137952 12700 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.142099 12674 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:21.142802 10121 debug-util.cc:398] Leaking SignalData structure 0x7b0800064540 after lost signal to thread 10291
I20240507 08:06:21.143573 12729 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:21.143855 10121 debug-util.cc:398] Leaking SignalData structure 0x7b08003afea0 after lost signal to thread 10365
I20240507 08:06:21.143924 12674 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:21.144308 10121 debug-util.cc:398] Leaking SignalData structure 0x7b0800305600 after lost signal to thread 10366
I20240507 08:06:21.141620 12750 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:21.144765 12750 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.144333 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.145265 12750 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.138362 12558 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.154016 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.149564 12558 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.155555 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.156210 10334 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:21.156435 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.157526 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.157752 12558 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.157837 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.158615 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.159639 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.150357 12700 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.160948 12700 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.162940 12700 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.153388 12674 raft_consensus.cc:1050] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:06:21.164882 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet db7c60767d74458094c1d86a49e87980 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:21.167726 12674 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.173496 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.176205 10341 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:21.174854 10334 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:21.177078 10334 raft_consensus.cc:595] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:21.150596 12767 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.178198 12767 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.178936 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.180168 12767 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.180936 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.180984 10156 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.182052 12729 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:21.182677 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.182642 12729 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.183650 10156 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:21.184342 10121 debug-util.cc:398] Leaking SignalData structure 0x7b08002265c0 after lost signal to thread 10967
I20240507 08:06:21.185089 12729 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:21.186342 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.189760 12730 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:21.194103 10943 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:21.190044 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.194329 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.194658 10185 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:21.194386 12730 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.195605 12730 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:21.201301 10156 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:21.202096 12749 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:21.191392 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.202476 12749 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.202874 12749 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:21.203615 12750 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.204648 12730 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.192184 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.192790 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.207002 10192 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:21.207417 12060 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.207844 12060 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.209401 12060 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.210572 12749 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.210976 12736 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.211555 12736 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.212496 10156 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:21.213383 12736 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.213958 12704 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.214251 12731 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:21.214485 12736 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.215169 12700 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.214483 12704 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.215888 12700 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.216980 12704 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.217487 12761 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.215013 12767 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.218107 12761 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.218405 12767 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.219226 12749 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.217600 12060 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.220042 12761 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.220314 12060 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.215051 12736 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.222128 12060 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.215329 10156 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.222834 12729 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.223201 10156 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.225322 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.225862 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.226523 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.227092 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.227510 12700 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.227700 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.220029 12767 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.228459 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.218039 12730 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.229446 12730 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:21.229843 12729 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.229846 12730 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.230909 12730 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:21.234076 10158 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:21.234611 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.235359 10232 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:21.235682 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.236183 12767 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:21.236538 12767 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.236898 12767 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:21.236793 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.237522 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.225908 12060 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:21.239027 12730 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.239687 12060 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.240494 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.241564 12730 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.241945 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:21.242331 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 254477us
I20240507 08:06:21.243191 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.243167 10306 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.244589 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240507 08:06:21.245376 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.246121 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:21.246738 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:21.246887 10907 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:21.248598 10907 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:21.249742 12754 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:21.250695 10306 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.251367 12681 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:21.251430 12754 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:21.251883 12681 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.252274 12681 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:21.253123 10907 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:21.249790 12727 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:21.254345 12727 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.254763 12727 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:21.255939 12758 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:21.258792 12736 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.259053 12681 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.260881 12681 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.262508 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:21.263131 10938 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.265892 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.266510 10260 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:21.267856 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.269165 12731 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:21.269555 12731 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.269942 12731 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.270048 10938 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:21.271145 10307 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:21.275105 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 52043.4us
I20240507 08:06:21.276006 12727 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.276952 12750 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.287186 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.313868 10939 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEARNER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:21.303424 10940 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:21.314213 10261 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:21.286737 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.286274 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.289386 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.287546 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.316998 10267 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:21.317126 10266 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:21.277446 12702 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:21.290479 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.318593 10259 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:06:21.318835 12702 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:21.286862 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.289920 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.279489 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:21.276871 12731 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.321959 12749 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:21.285701 10907 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:21.323539 12731 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.323629 12769 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:21.324128 12731 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:21.320484 10265 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:21.281157 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.325109 12749 raft_consensus.cc:3032] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:21.325594 12749 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:21.326560 12749 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.327863 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.328344 12767 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.329421 10262 raft_consensus.cc:2379] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 6: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:21.329651 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.279011 10232 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:21.330389 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.331912 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.333379 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.334826 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.241485 12060 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.336012 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.336584 10185 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:21.276281 10156 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.327901 12749 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.281886 12730 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:21.339058 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:21.339761 10938 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.339195 12730 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.341399 12730 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:21.342039 12674 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:21.342310 12727 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.343523 12761 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.343850 12767 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.344050 12761 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:21.341511 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 65316.4us
W20240507 08:06:21.346890 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 96293us
I20240507 08:06:21.348945 10306 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.353826 10306 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.344360 12767 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:21.360316 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 81408us
I20240507 08:06:21.370205 10306 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.371417 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.372072 10186 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:21.372431 12761 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.349371 12674 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.385816 12674 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.368080 12767 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.388934 12767 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:21.389199 10938 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:21.389855 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.390995 12681 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:21.390938 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.390588 10158 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:21.391599 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.394451 12681 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.395128 10263 raft_consensus.cc:2379] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:21.394140 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.394963 12681 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:21.397459 12736 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:21.397730 12060 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.398030 12736 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.398183 12060 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.399794 12060 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.399820 12736 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.368736 12700 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:21.400628 12700 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.400974 12700 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:21.401892 12768 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.402379 12768 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.404315 12768 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.367854 12750 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:21.394959 12729 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:21.411201 10334 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:21.411489 12750 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.411612 12729 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:21.412164 12750 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:21.412776 10156 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.413887 12722 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:21.414255 12722 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.414613 12722 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:21.415933 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.416646 10341 raft_consensus.cc:2379] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:21.416513 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.417116 10336 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:21.417415 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.420217 12748 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.420681 12748 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.421984 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.422191 10334 raft_consensus.cc:1226] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:21.422535 10333 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:21.357764 12702 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.422535 12748 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.426203 10335 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:21.412786 12729 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.410364 10232 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:21.429548 12767 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:21.429896 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.431464 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.432009 10156 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:21.393714 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:21.431334 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 61210.6us
I20240507 08:06:21.433851 12749 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:21.429924 12767 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:21.434424 12767 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.434969 12749 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:21.437109 10336 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:21.434439 12748 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:21.438599 12748 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:21.439045 12748 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.439671 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.439808 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.438908 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.441129 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.441485 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.441926 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.441937 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.438053 10333 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:21.445953 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.447126 10267 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:21.447466 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.449440 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.450598 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.452343 10340 raft_consensus.cc:1226] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:21.453981 12748 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.454417 12730 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:21.455794 12748 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.456563 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.463312 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.463845 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.456761 12730 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:21.457964 10186 raft_consensus.cc:2454] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:21.458791 10156 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.461059 10231 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:21.470716 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.466281 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.470366 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.464984 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.465342 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.466461 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.486986 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.484117 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.489310 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.490694 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.493114 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.494089 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.487527 10156 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:21.497385 10232 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:21.494139 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.495287 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.515568 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.520695 10183 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:21.515187 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.521154 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.520483 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.521533 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.519601 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.515836 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.494712 12764 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:21.462023 12722 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.520586 12768 raft_consensus.cc:2781] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:21.532753 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.533231 12768 raft_consensus.cc:686] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:21.534026 12722 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.525775 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.518618 12700 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.535344 12764 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:21.536275 12764 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:21.536902 12700 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:21.537035 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.537837 10258 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:21.538071 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.539269 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.539744 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.540289 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.541107 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.541625 10334 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:21.542259 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.542869 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.543452 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.528349 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.528795 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.529376 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.522083 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.531379 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:21.543519 10264 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:21.496994 12730 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:21.591492 10335 raft_consensus.cc:1226] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:21.544263 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.595371 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.596308 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.592805 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.597406 10267 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:21.547583 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.598289 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.598793 12730 raft_consensus.cc:686] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:21.599296 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.599648 12730 consensus_queue.cc:237] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.600325 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.592495 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.601285 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.597326 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.602702 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.609797 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.611279 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.605918 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:21.606267 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 76399.9us
I20240507 08:06:21.606768 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.608691 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.604677 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.596664 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.609797 10087 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 7 to 8, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 8 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:21.615979 10192 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:21.620739 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.620980 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.621829 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.621958 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.623106 10339 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:21.624117 10339 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:21.624796 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:21.487239 12674 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.626760 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.704104 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.713384 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.715436 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.705930 10942 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:21.514915 12729 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:21.710896 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.702389 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.627395 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.650278 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.713768 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:21.682345 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 55906.5us
I20240507 08:06:21.679780 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.678032 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.638401 12558 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20240507 08:06:21.728701 10233 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:21.729492 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:21.734676 10233 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:21.735255 12060 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:21.735929 10233 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:21.743197 10186 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:21.789829 10233 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:21.796180 12704 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:21.796917 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.802140 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:21.499347 12767 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.742954 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.788889 12060 consensus_queue.cc:1046] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:21.796097 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:21.704180 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:21.801431 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:21.808310 10188 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:21.808691 10304 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:21.614781 12681 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.813540 10304 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:21.814147 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.733698 12736 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:21.819748 10304 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:21.826661 12465 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:21.728344 12702 consensus_queue.cc:1046] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20240507 08:06:21.830833 12060 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:21.832233 12060 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:21.817696 12679 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:21.833709 10262 raft_consensus.cc:2379] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 8: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:21.832796 12060 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:21.838017 12060 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:21.836899 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:21.841629 10332 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:21.534060 12768 consensus_queue.cc:237] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.824035 12450 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:21.817451 12751 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:21.817838 12763 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:21.844676 10260 raft_consensus.cc:1226] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:21.845490 12765 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20240507 08:06:21.862278 12748 raft_consensus.cc:979] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:21.862110 10233 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:21.868068 12450 raft_consensus.cc:3032] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:21.845786 12729 raft_consensus.cc:686] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:21.877549 12729 consensus_queue.cc:237] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.812685 12750 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:21.868479 12450 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:21.922899 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 96725us
I20240507 08:06:21.876161 12736 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:21.902638 10906 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:21.961351 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 126894us
I20240507 08:06:21.962679 10906 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:21.963270 10304 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:21.964306 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:21.958288 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 75648.4us
I20240507 08:06:21.959303 10333 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:21.982342 12758 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:21.969796 10906 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.004892 10906 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.005789 12558 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240507 08:06:22.007550 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 284541us
W20240507 08:06:22.008230 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 183990us
I20240507 08:06:22.009716 12586 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:22.014446 10261 raft_consensus.cc:1226] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:22.014623 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 219381us
I20240507 08:06:21.959878 12679 raft_consensus.cc:3032] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:22.025336 10192 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:21.957958 12755 raft_consensus.cc:3032] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:21.958020 12748 raft_consensus.cc:1067] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:22.028087 10190 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:22.028797 12748 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.029306 12748 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240507 08:06:21.968246 12776 log.cc:924] Time spent T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.093s user 0.000s sys 0.004s
I20240507 08:06:22.025651 12679 raft_consensus.cc:729] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:21.980237 12754 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.027611 10186 raft_consensus.cc:1226] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:22.026990 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:22.029683 12755 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:22.018965 12779 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.041890 12767 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:22.042989 12681 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:22.044229 12679 consensus_queue.cc:260] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.046414 12679 raft_consensus.cc:3037] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:22.022699 12674 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:22.060190 12767 consensus_queue.cc:1046] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20240507 08:06:22.018419 12450 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.029805 12768 consensus_queue.cc:1046] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20240507 08:06:22.064314 12755 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.065311 12768 raft_consensus.cc:3032] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:22.065696 12768 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.066040 12755 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:22.066462 12768 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.064357 12450 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.067692 12768 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.070216 12750 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:22.071422 12767 consensus_queue.cc:1225] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20240507 08:06:22.072634 12783 raft_consensus.cc:3032] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:22.072948 12783 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.073709 12783 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.035472 12763 consensus_queue.cc:1046] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240507 08:06:22.076246 12763 raft_consensus.cc:3032] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:22.076610 12763 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:22.094180 12781 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.122s user 0.003s sys 0.000s
I20240507 08:06:22.093688 10334 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:22.097610 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.101044 10183 raft_consensus.cc:1226] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:22.077420 12763 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.110253 12763 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.121779 10192 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55820
I20240507 08:06:22.122349 10192 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:22.122680 10192 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.123219 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:22.125867 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:22.125381 10185 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240507 08:06:22.138298 10938 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 6 LEARNER]: Advancing to term 7
I20240507 08:06:22.138357 10333 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:22.147845 10192 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.150274 10189 raft_consensus.cc:1226] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
W20240507 08:06:22.150532 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 118138us
W20240507 08:06:22.151196 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 187032us
I20240507 08:06:22.157795 10304 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:22.154531 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.158526 12759 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:22.161545 10304 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:22.162464 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 145482us
I20240507 08:06:22.163285 10192 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:22.163713 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.164414 12586 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:22.164788 12586 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:22.165153 12586 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:22.164239 12681 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:22.166368 12681 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:22.166708 12681 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.167886 12782 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.170320 10304 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.171312 12725 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.171581 12782 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:22.162271 12730 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:22.172603 12730 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:22.172959 12730 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.173355 12725 consensus_queue.cc:1046] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:22.173156 10155 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.173740 12586 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.175793 12759 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.176247 12725 raft_consensus.cc:3032] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:22.178185 12725 raft_consensus.cc:729] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:22.178930 12725 consensus_queue.cc:260] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.180193 12725 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.180379 12730 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.181767 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.182607 12793 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.183858 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:22.177313 10155 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:22.185613 12586 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:22.185488 10304 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.186455 10155 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.175864 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.189357 10155 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:22.191172 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.193105 12750 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.193719 12759 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:22.193851 12778 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:22.194211 12759 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:22.194481 12778 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:22.194588 12759 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.194875 12778 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.195549 10939 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240507 08:06:22.202008 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: true dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:22.202569 10939 raft_consensus.cc:3037] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.203940 10089 catalog_manager.cc:5484] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:22.206393 12759 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.193718 12586 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:22.209764 12586 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:22.210148 12586 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:22.190223 10155 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.211534 10304 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.209189 12793 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20240507 08:06:22.218868 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 185713us
I20240507 08:06:22.217736 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.225857 12789 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.244781 12794 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:22.243829 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.245363 12794 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:22.245275 12790 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240507 08:06:22.245893 12794 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:22.247308 10304 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.220880 12783 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:22.249794 12750 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.250341 12750 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.250738 10185 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:22.250905 10187 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.254846 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.256991 12750 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:22.229108 12793 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:22.257320 10187 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:22.227115 10942 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20240507 08:06:22.241989 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 51853.6us
I20240507 08:06:22.237326 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.236794 10337 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:22.259231 10304 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.259316 10943 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:22.225251 10939 raft_consensus.cc:2454] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:22.264597 10184 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:22.265064 10155 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.268739 10156 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:22.244560 12789 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.271816 12790 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.272171 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:22.237269 12759 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:22.228757 12778 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.225991 10938 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:22.274659 12778 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:22.274904 12798 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:22.277637 10187 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:22.235651 12586 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.285012 12586 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:22.292006 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.257674 12730 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:22.257632 12793 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:22.290863 12767 consensus_queue.cc:1046] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20240507 08:06:22.284577 12783 consensus_queue.cc:1046] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20240507 08:06:22.301410 12767 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:22.301792 12767 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.302556 12768 raft_consensus.cc:3032] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:22.302963 12768 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.302484 12767 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.303709 12767 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:22.303740 12768 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:22.304809 12768 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.300565 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:22.307735 10155 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.305706 12733 raft_consensus.cc:2781] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:22.312979 10304 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.225739 12681 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.347721 12465 consensus_queue.cc:1046] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:22.362744 12465 raft_consensus.cc:3032] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:22.363163 12465 raft_consensus.cc:729] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:22.306967 12782 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20240507 08:06:22.290681 12786 consensus_queue.cc:1046] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20240507 08:06:22.293921 12798 consensus_queue.cc:1046] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20240507 08:06:22.350136 12679 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:22.354880 12794 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.295842 12450 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.315268 12704 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20240507 08:06:22.306561 12765 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.358749 10340 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:22.386085 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.359632 12681 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:22.360839 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 64512.3us
I20240507 08:06:22.363629 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:22.366544 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:22.368400 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:22.370707 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 63117.4us
I20240507 08:06:22.364027 12465 consensus_queue.cc:260] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.394110 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.394281 10155 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.394788 12465 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:22.360124 12793 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.387610 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:22.377095 12730 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.389423 10337 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:22.393384 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.388053 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.415030 10155 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.416580 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.418357 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.379580 12733 raft_consensus.cc:686] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:22.427348 10231 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.428421 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.422441 12060 consensus_queue.cc:1046] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20240507 08:06:22.427837 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.432130 12558 raft_consensus.cc:3032] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:22.432417 12558 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:22.400583 12798 consensus_queue.cc:1225] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240507 08:06:22.435485 10304 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.436641 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.438688 10304 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.439415 12558 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.417910 12790 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.443305 12558 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.444918 10304 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.380136 12783 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20240507 08:06:22.448139 10333 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:22.429996 12764 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.449213 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:22.431205 12736 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:22.398608 12703 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20240507 08:06:22.458149 12733 consensus_queue.cc:237] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.443032 12702 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.441010 12798 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.442901 12748 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.447873 12782 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.471496 12780 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.474644 10334 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:22.451208 12450 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20240507 08:06:22.498083 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 71464.8us
W20240507 08:06:22.493067 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 64718.9us
I20240507 08:06:22.426649 12769 consensus_queue.cc:1046] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20240507 08:06:22.503264 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.517637 12763 raft_consensus.cc:3032] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:22.518102 12763 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.518604 12793 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:22.518985 12763 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.519917 12794 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:22.520357 12763 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:22.533071 12786 raft_consensus.cc:3032] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:22.533627 12786 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:22.534742 12786 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.536485 12786 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.536741 12730 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.536553 12761 raft_consensus.cc:3032] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:22.537842 12761 raft_consensus.cc:729] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.538579 12761 consensus_queue.cc:260] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.539487 10184 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:22.540030 10184 raft_consensus.cc:595] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:22.536574 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 121701us
W20240507 08:06:22.548154 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 99061.6us
W20240507 08:06:22.548431 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 45267.4us
I20240507 08:06:22.549824 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:22.552439 10231 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.553422 10192 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:22.554858 10185 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:22.558109 10155 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.557891 10231 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:22.550583 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 105773us
I20240507 08:06:22.558969 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.559769 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.560901 10231 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.563089 12586 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.563696 10231 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.536553 12811 raft_consensus.cc:3032] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:22.547434 12704 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.570806 10190 raft_consensus.cc:1226] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:22.569154 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.573062 12586 raft_consensus.cc:3037] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:22.573113 12060 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.574115 10332 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:22.574419 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.574677 10332 raft_consensus.cc:595] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:22.575356 10332 raft_consensus.cc:603] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:22.571836 12558 consensus_queue.cc:1046] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20240507 08:06:22.575433 10231 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.560539 12674 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:22.572578 12700 raft_consensus.cc:1050] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:06:22.585249 12700 raft_consensus.cc:1054] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Unable to promote non-voter 5d6656a616b747d393ff3de5328d2540: Illegal state: Leader has not yet committed an operation in its own term
I20240507 08:06:22.586048 12780 consensus_queue.cc:1046] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:22.586556 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.574575 12761 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:22.562144 12703 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.581305 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.572150 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:22.581877 10260 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240507 08:06:22.582984 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.599241 10264 raft_consensus.cc:2379] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:22.585249 12704 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.588485 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.567695 12759 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.589726 10262 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20240507 08:06:22.581229 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.609946 12558 raft_consensus.cc:3032] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:22.594918 12809 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.572129 12681 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20240507 08:06:22.610253 12558 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:22.602489 10086 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 7 to 8, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 8 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:22.577710 10231 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.606088 10332 raft_consensus.cc:1261] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:22.569919 12783 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.611687 10086 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:22.615411 10231 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.618717 10086 catalog_manager.cc:5484] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 2 to 3, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 3 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:22.620317 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.620554 12703 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.570077 12798 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.622404 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.622522 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.568043 12811 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:22.634948 12812 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20240507 08:06:22.635092 12811 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.636433 12811 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:22.623653 10231 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.631577 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:22.567785 12729 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20240507 08:06:22.641110 12586 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.642168 10231 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.647784 12731 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:22.654273 12765 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.655431 10332 raft_consensus.cc:1226] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:22.634179 12798 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.640604 12783 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:22.666612 10231 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.666895 12809 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.667569 12809 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.667742 10231 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.668125 10155 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.668841 10231 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.669968 12798 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:22.670516 12798 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.670473 12558 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.671936 12558 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:22.672015 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.672885 12783 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.673714 12730 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:06:22.674167 10231 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.593271 12810 consensus_queue.cc:1046] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20240507 08:06:22.675228 12703 raft_consensus.cc:3032] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:22.675590 12703 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.630498 12704 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.678391 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.687186 10259 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:22.687902 10231 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.689436 12060 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.689778 10231 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.691097 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.692592 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.692963 12060 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.697777 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.700613 10087 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59936 (request call id 206) took 1183 ms. Trace:
I20240507 08:06:22.703289 12674 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:22.694312 12809 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.703852 12703 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.694442 12764 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:22.712693 10185 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:22.689007 12700 consensus_queue.cc:1046] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20240507 08:06:22.714234 12748 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:22.714679 10262 raft_consensus.cc:1261] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:22.714942 12700 raft_consensus.cc:3032] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:22.715194 12748 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:22.715324 12700 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.715651 12748 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:22.716531 12748 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.728461 12748 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:22.713392 12807 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:22.694202 12755 consensus_queue.cc:1046] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240507 08:06:22.737799 12807 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:22.738760 12755 raft_consensus.cc:3032] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:22.738608 12807 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.686069 12783 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.739938 12807 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.726598 12704 raft_consensus.cc:1050] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
W20240507 08:06:22.741063 12704 raft_consensus.cc:1054] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Unable to promote non-voter 5d6656a616b747d393ff3de5328d2540: Illegal state: Leader has not yet committed an operation in its own term
I20240507 08:06:22.718420 12725 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:22.741634 12725 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:22.742405 12725 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.742676 12674 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.743593 12725 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
W20240507 08:06:22.743718 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 46055.3us
I20240507 08:06:22.739109 12755 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:22.745198 10155 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.728129 12700 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.746105 12778 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:22.746767 12700 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:22.746713 12778 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:22.748049 12778 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:22.698957 12798 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240507 08:06:22.730471 12817 log.cc:924] Time spent T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20240507 08:06:22.700951 10087 rpcz_store.cc:276] 0507 08:06:21.517239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:21.531671 (+ 14432us) service_pool.cc:225] Handling call
0507 08:06:22.700551 (+1168880us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:21.854695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:21.857409 (+ 2714us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:21.857445 (+ 36us) write_op.cc:435] Acquired schema lock
0507 08:06:21.857478 (+ 33us) tablet.cc:660] Decoding operations
0507 08:06:21.857801 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:21.857853 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:21.857879 (+ 26us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:06:21.858048 (+ 169us) tablet.cc:699] Row locks acquired
0507 08:06:21.858064 (+ 16us) write_op.cc:260] PREPARE: finished
0507 08:06:21.858186 (+ 122us) write_op.cc:270] Start()
0507 08:06:21.866667 (+ 8481us) write_op.cc:276] Timestamp: P: 1715069181858147 usec, L: 0
0507 08:06:21.868836 (+ 2169us) op_driver.cc:331] REPLICATION: starting
0507 08:06:22.035935 (+167099us) log.cc:842] Serialized 536 byte log entry
0507 08:06:22.538780 (+502845us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b54001a1920. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90dbde83 00007fcf90dc18b5 00007fcf90dc1849 00007fcf90dc17d9 00007fcf90dc17a1 00007fcf90dc0a9d 00007fcf97164224 00007fcf97164059 00007fcf8defd236 00007fcf8df00691
0507 08:06:22.538990 (+ 210us) op_driver.cc:444] REPLICATION: finished
0507 08:06:22.584417 (+ 45427us) write_op.cc:301] APPLY: starting
0507 08:06:22.605634 (+ 21217us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:22.634010 (+ 28376us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:22.637963 (+ 3953us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:22.644277 (+ 6314us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:22.644456 (+ 179us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:06:22.644507 (+ 51us) write_op.cc:312] APPLY: finished
0507 08:06:22.644698 (+ 191us) log.cc:842] Serialized 23 byte log entry
0507 08:06:22.644877 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:22.644953 (+ 76us) write_op.cc:454] Released schema lock
0507 08:06:22.644994 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:22.645250 (+ 256us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:22.645280 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":87,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":29460,"apply.run_cpu_time_us":1943,"apply.run_wall_time_us":75615,"num_ops":1,"prepare.queue_time_us":214893,"prepare.run_cpu_time_us":17703,"prepare.run_wall_time_us":339342,"raft.queue_time_us":370776,"raft.run_cpu_time_us":10963,"raft.run_wall_time_us":117343,"replication_time_us":664428,"spinlock_wait_cycles":25894144,"thread_start_us":425362,"threads_started":4,"wal-append.queue_time_us":147712}]]}
I20240507 08:06:22.739095 12703 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:22.758306 10088 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20240507 08:06:22.745151 12755 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:22.760813 12755 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:22.785305 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 91377.2us
I20240507 08:06:22.786666 10231 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.788353 12060 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.789484 10231 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.791620 12060 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.793193 12748 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:22.793301 12060 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:22.815042 12060 raft_consensus.cc:3032] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:22.815385 12060 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.816073 12060 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.817292 12060 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:22.798084 12700 consensus_queue.cc:1046] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:22.816723 12748 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:22.820490 12700 raft_consensus.cc:3032] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:22.820874 12700 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.821684 12700 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.822759 12700 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:22.800125 12703 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:22.801057 10231 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:22.811384 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 66276.5us
I20240507 08:06:22.826351 10337 raft_consensus.cc:1261] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:22.826371 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.789500 10086 catalog_manager.cc:5484] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 6, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:22.829885 12798 raft_consensus.cc:2781] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:22.830453 12798 raft_consensus.cc:686] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.830591 12748 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:22.831175 10333 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:22.831351 12798 consensus_queue.cc:237] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.832226 12731 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:22.832444 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:22.833719 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:22.834990 10266 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:22.839402 10266 raft_consensus.cc:595] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.839761 10266 raft_consensus.cc:603] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:22.840960 10155 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.841784 12731 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240507 08:06:22.843752 12729 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:22.841748 12703 consensus_queue.cc:1046] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240507 08:06:22.842206 10306 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.850965 10333 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:22.851487 10333 raft_consensus.cc:595] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:22.854163 12824 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:22.854676 12824 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:22.855583 12824 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.856964 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:22.857401 12824 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:22.859815 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240507 08:06:22.861542 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:22.865772 12729 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:22.866283 12729 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:22.867079 12729 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:22.867729 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:22.878966 12731 raft_consensus.cc:979] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:22.877642 12674 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240507 08:06:22.879467 12731 raft_consensus.cc:1067] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:22.895161 10262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:22.895849 10262 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:22.896236 10262 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:22.838740 12700 consensus_queue.cc:1035] T d26fdad3816d470a8a7537d14838ac68 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20240507 08:06:22.877002 12748 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20240507 08:06:23.033951 12815 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:06:23.026736 10259 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 5 to 6, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.040601 10089 catalog_manager.cc:5484] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 7 to 8, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 8 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:23.039142 10262 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.057857 10258 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:23.060890 10262 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:23.052356 12781 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.106s user 0.006s sys 0.000s
I20240507 08:06:23.065825 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:23.074031 10189 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:23.074358 10189 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:23.078532 10189 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.079715 10189 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
W20240507 08:06:23.096217 12825 log.cc:924] Time spent T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:06:23.094331 10189 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:23.114051 10232 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.121430 10940 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:23.125320 10337 raft_consensus.cc:1261] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:23.069092 12793 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:23.151628 12793 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.160948 12787 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:23.161484 12787 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.162863 12783 raft_consensus.cc:1050] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: attempting to promote NON_VOTER 5d6656a616b747d393ff3de5328d2540 to VOTER
I20240507 08:06:23.163497 12787 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:23.164429 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:23.165050 10262 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:23.166126 10304 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.136418 12703 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:23.167057 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:23.169037 10337 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:23.171953 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.175655 10185 raft_consensus.cc:2379] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:23.182427 10337 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:23.185971 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.180658 12703 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:23.205963 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:23.206724 10266 raft_consensus.cc:2379] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 8: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:23.191308 12681 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:23.208343 10304 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:23.175194 12674 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20240507 08:06:23.207265 12793 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:23.165398 12783 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.208469 12681 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:23.215778 12780 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:23.226075 12681 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:23.237731 10940 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEARNER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:23.240345 10334 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:23.242393 12764 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:23.251241 10086 catalog_manager.cc:5484] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 5, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:23.255012 10086 catalog_manager.cc:5484] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:23.256016 10086 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 7 to 8, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 8 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:23.259756 10187 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:23.220786 12749 consensus_queue.cc:1035] T ffd4c65b5ad04f48916c3e359f7e45c9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20240507 08:06:23.246399 12821 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
W20240507 08:06:23.265612 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 55444.7us
I20240507 08:06:23.271730 12681 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.273818 12681 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:23.275192 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:23.284883 12783 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:23.291265 10265 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:23.293295 12766 raft_consensus.cc:979] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:23.281802 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:23.296281 10304 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.297178 10187 raft_consensus.cc:2379] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
W20240507 08:06:23.300377 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 52044.1us
I20240507 08:06:23.246917 12703 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.300284 12748 raft_consensus.cc:1067] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:23.307814 10187 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:23.312245 12821 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:23.317000 10265 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20240507 08:06:23.311400 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 55720.7us
I20240507 08:06:23.322736 10337 raft_consensus.cc:1261] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:23.308514 12787 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:23.328320 12787 raft_consensus.cc:686] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:23.329169 12787 consensus_queue.cc:237] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.329198 12060 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Committing config change with OpId 7.8: config changed from index 5 to 8, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.331526 12674 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:23.332654 12824 consensus_queue.cc:1035] T 10eece2dd4a64e3aa99d91fadc43df89 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:23.340076 12780 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:23.351692 10266 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:23.354542 10334 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:23.356443 10262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42924
I20240507 08:06:23.356966 10262 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:23.357376 10262 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:23.357028 10190 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.358314 12722 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:23.364302 10262 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.373730 10938 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 5 to 8, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.375780 10183 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 5 to 8, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.367573 12722 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20240507 08:06:23.407887 10333 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:23.411731 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:23.412330 10333 raft_consensus.cc:3032] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:23.408128 10341 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Committing config change with OpId 7.8: config changed from index 5 to 8, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.412613 10333 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:23.420025 10262 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:23.416265 10333 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.440078 10333 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:23.456476 10333 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:23.457962 10231 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.432708 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:23.427367 12759 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20240507 08:06:23.477936 10086 catalog_manager.cc:5484] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:23.489750 10332 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:23.495258 10088 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: config changed from index 5 to 8, 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } attrs { replace: true } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.467406 12703 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:23.500167 12704 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:23.504840 10938 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:23.478973 12674 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:23.479912 10939 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:23.526717 12704 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:23.529922 12731 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:23.528127 12703 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:23.532433 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.533128 10189 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:23.533246 12703 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.538638 10232 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.540728 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.541318 10336 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:23.544886 12786 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:23.545329 12786 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.539348 12824 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:23.546844 12824 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:23.547185 12786 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.547256 12818 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:23.550567 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.528573 12674 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.554066 12674 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:23.555200 10189 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:06:23.557351 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.557746 10908 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:23.539829 12704 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.563076 12840 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:23.563443 12840 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:23.563756 12840 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:23.564271 12787 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:23.564764 12787 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.567836 12787 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:23.570477 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.571257 10190 raft_consensus.cc:2379] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:23.573823 10087 catalog_manager.cc:5484] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 7 to 8, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 8 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:23.575654 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:23.547150 12824 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:23.583760 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.587752 10336 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:06:23.582749 12783 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:23.601799 12783 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.603662 12783 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.608644 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.609476 10185 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:23.607981 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.610622 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.615609 10265 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.595129 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:23.626322 10307 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:23.627379 12681 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:23.636669 10939 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:23.637810 12783 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:23.638276 12783 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:23.638592 12783 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:23.610513 10334 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:23.641685 10943 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240507 08:06:23.643364 10189 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240507 08:06:23.643913 12704 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:23.654454 12764 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:23.658715 12840 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.659968 10934 raft_consensus.cc:2379] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:23.661618 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:23.662264 10190 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:23.662421 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:23.667676 10158 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:23.661597 12840 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:23.698275 12847 log.cc:924] Time spent T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20240507 08:06:23.695130 12704 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:23.722673 12783 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.724692 12783 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.725723 12704 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.690922 12824 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:23.737939 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:23.738529 10340 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:23.741132 10190 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:23.742996 10908 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:23.745194 12840 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:23.745748 12840 raft_consensus.cc:686] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:23.746815 12840 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:23.748883 12681 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:23.749495 10334 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:23.755676 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:23.755085 10943 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 8 to 9, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.756218 10190 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:23.760460 10332 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:23.761979 12060 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:23.763713 10340 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:23.767328 10231 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.765985 10190 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:23.773541 10334 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:23.773638 10265 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:23.776140 12060 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:23.776504 12060 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:23.777027 12060 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.788488 10190 raft_consensus.cc:1261] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240507 08:06:23.781360 12759 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:23.796054 12759 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.798334 12759 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.795326 12748 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20240507 08:06:23.799854 12704 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:23.742753 12818 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:23.803939 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:23.804523 10340 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 6
I20240507 08:06:23.805316 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.806038 10189 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:23.798694 12824 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.812579 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.818620 10334 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:23.814999 10340 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:23.821111 10231 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.822031 12824 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:23.822513 12824 raft_consensus.cc:686] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:23.819960 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:23.823830 10185 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:23.809274 12731 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:23.831785 12794 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.751407 12821 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Committing config change with OpId 7.9: config changed from index 8 to 9, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.831805 12703 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:23.851490 12703 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.861331 12703 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.842273 12731 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.875939 12731 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.847155 12824 consensus_queue.cc:237] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
W20240507 08:06:23.913213 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 134531us
I20240507 08:06:23.928645 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:23.930233 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.930334 10266 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:23.931613 10334 raft_consensus.cc:2379] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 6: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:23.933557 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.937681 10332 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:23.940008 10074 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a181f4ae4d342129ad262e5545debc9 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:23.943116 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:23.944008 10940 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:23.944082 12681 consensus_queue.cc:1035] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:23.951081 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:23.952075 10265 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:23.959127 12749 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:23.966017 10183 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:23.970724 10187 raft_consensus.cc:2932] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Committing config change with OpId 7.9: config changed from index 8 to 9, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.979189 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:23.972483 12749 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:23.974645 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:23.986095 12749 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:23.991461 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:23.992544 10336 raft_consensus.cc:2379] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:23.977027 10940 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:23.999969 12759 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:24.000461 12759 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:24.000847 12759 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:24.007458 12787 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.008044 12787 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.010684 12787 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.011974 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.012912 10183 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:23.993283 10155 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:23.994055 10089 catalog_manager.cc:5484] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:24.025753 10306 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.031237 12681 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:24.031742 12681 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.032145 12681 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:24.015128 12764 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20240507 08:06:24.035584 12807 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.036399 12807 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.039227 12807 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.040472 12748 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.040628 12766 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.041327 12766 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:24.043797 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48326.5us
I20240507 08:06:24.041340 12748 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.050884 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:24.050285 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 57241.9us
I20240507 08:06:24.050824 12749 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:24.053848 12749 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:24.053849 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:24.054406 12749 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.054867 10934 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:24.060690 10307 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.060614 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.061640 10192 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:24.061817 12780 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:24.062341 12780 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.062691 12780 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.062952 10306 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.065613 12787 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:24.066025 12787 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.066397 12787 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.044143 12766 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.053035 12748 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.067230 10334 raft_consensus.cc:2932] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Committing config change with OpId 5.7: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:24.061642 10183 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:24.072252 10306 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.073096 12748 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:24.073487 12748 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.073923 12748 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:24.049197 10231 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:24.077973 12764 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:24.078362 12764 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.078769 12764 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.079458 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:24.057222 10233 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:24.084988 12759 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.097796 12674 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:24.098469 12674 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.100857 12674 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.101644 10185 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:24.117944 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.118880 10332 raft_consensus.cc:2379] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 8: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:24.124708 12780 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.128173 10076 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 03d2e621edea41ae97e79e36e4dc0365 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240507 08:06:24.129880 12858 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.130676 12766 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.176160 12759 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.179818 12818 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:24.180559 12818 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.182824 12818 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.165865 12858 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.185993 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.168557 10087 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: config changed from index 8 to 9, VOTER 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) evicted. New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:24.186511 10187 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:24.190608 12824 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:24.191219 12824 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.209220 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.210420 10333 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:24.210902 10087 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:24.212095 10336 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:24.215301 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:24.216672 12704 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:24.233902 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.234879 10187 raft_consensus.cc:2379] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:24.170979 12766 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.237102 12766 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.239233 12824 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.240092 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.241874 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.242357 10340 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:24.242630 10333 raft_consensus.cc:2379] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:24.245155 12821 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.247876 12703 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:24.185387 12858 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.217957 12704 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.256618 12704 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
W20240507 08:06:24.261924 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 267592us
I20240507 08:06:24.228309 12749 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.268146 12780 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.268280 12748 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.269040 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:24.270387 12749 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.271392 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:24.274711 10940 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.280442 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.281176 10262 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:24.283584 12870 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.283696 10086 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: config changed from index 6 to 7, VOTER bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) evicted. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:24.245187 12787 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.284209 12870 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.253141 12821 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.289810 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.290606 10266 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:24.290685 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.291404 10259 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:24.291589 12821 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.291654 10934 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:24.292932 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.293669 10265 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:24.294118 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.304149 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.315846 12748 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.317694 12866 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.318255 12866 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.319519 12681 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.320317 12866 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.320699 10333 raft_consensus.cc:2354] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:24.321627 12860 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:24.322223 12860 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:24.322705 12860 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.324528 12681 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.330827 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.331459 10192 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:24.336823 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.337366 10266 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.339497 10156 leader_election.cc:398] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:24.339908 10156 leader_election.cc:401] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240507 08:06:24.341102 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.341986 10265 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:24.345654 10187 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:24.318454 12870 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.350932 12060 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:24.351625 12060 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.352748 10266 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:24.348632 10940 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:24.355594 10232 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:24.359584 12060 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.367118 10187 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:24.367729 10307 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.369222 12681 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:24.369807 12681 raft_consensus.cc:686] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:24.371313 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 72860.1us
I20240507 08:06:24.371075 12681 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.372356 12672 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.372869 12672 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.362783 12870 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.374662 12870 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.375238 12704 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.376591 12870 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.379205 12787 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.380300 12672 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.382629 10340 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:24.369699 12786 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:24.383718 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.384821 10266 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:24.385962 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.386724 10262 raft_consensus.cc:2354] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9.
I20240507 08:06:24.389058 12759 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:24.394336 12749 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:24.395099 12749 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.395956 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.396633 12704 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.397056 10333 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:24.397758 10336 raft_consensus.cc:1261] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:24.399302 12749 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.399528 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.401001 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.419076 10333 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:24.418042 10336 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:24.363848 12821 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240507 08:06:24.425870 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 126137us
I20240507 08:06:24.428562 12821 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.440237 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.440963 10192 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:24.441900 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.442548 10187 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:24.446599 10323 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a181f4ae4d342129ad262e5545debc9 with delete_type TABLET_DATA_TOMBSTONED (TS 88a26dac37fa4243b8c2b3d47f403488 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39708
I20240507 08:06:24.448004 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:24.440387 12749 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:24.448664 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.448815 12749 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.449050 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.449882 10189 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 9
I20240507 08:06:24.450892 12749 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.422627 12786 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20240507 08:06:24.442478 12821 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:24.453776 12872 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.454491 12872 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.449326 10192 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:24.458431 12060 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:24.458813 12060 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:24.459208 12060 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:24.428148 12704 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20240507 08:06:24.462249 12733 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.462646 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.462853 12733 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.465171 12733 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.440114 12783 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20240507 08:06:24.470458 12869 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:24.471184 12869 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.462245 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.474964 10341 raft_consensus.cc:2379] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:24.478302 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.479354 10190 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 7: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:24.480501 10183 raft_consensus.cc:1261] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:24.481217 12872 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.481106 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:24.482167 10940 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 3.
I20240507 08:06:24.482213 12764 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.484470 10158 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:24.440634 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:24.485165 12764 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.485366 12875 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:24.485672 10934 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:24.485845 12875 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.486240 12875 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:24.486145 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.497236 12869 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:24.502627 10189 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 9.
I20240507 08:06:24.509090 10158 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:24.516561 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.517554 10340 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:24.517324 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.518174 10189 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.519650 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.520223 10341 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.521502 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.518834 10333 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 9.
I20240507 08:06:24.525346 12794 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:24.525708 10336 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:06:24.525810 12794 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.526316 12794 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.526641 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.530005 10232 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:24.531333 12703 consensus_queue.cc:1035] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:24.542162 10189 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:24.551555 10907 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.551106 12794 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.552404 12872 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:24.552780 12872 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.552738 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.553344 12872 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.564167 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.565690 12794 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.569525 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:24.576741 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:24.577343 10934 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.573297 10187 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:24.578434 12783 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:24.588460 12783 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:24.593952 10262 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:24.561229 12821 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.585112 12872 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.601231 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.602288 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.602902 10183 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:24.599761 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.603732 10259 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:24.593701 12703 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:24.613174 10088 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 6, leader changed from 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:24.615577 10263 raft_consensus.cc:2354] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8.
I20240507 08:06:24.598943 10934 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:24.621831 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.572999 12860 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.622608 10183 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.589370 12783 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.629642 12821 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.623555 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.635661 10259 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 2.
I20240507 08:06:24.634858 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.636868 10340 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:24.641202 10266 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:24.644482 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.645257 10187 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 8
I20240507 08:06:24.619184 12703 raft_consensus.cc:686] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:24.641112 10158 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:24.651243 10304 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:24.657286 10940 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:24.661736 12749 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:24.664242 12764 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:24.665824 12783 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:24.673197 12759 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:24.673399 12703 consensus_queue.cc:237] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.673542 12783 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } attrs { replace: true } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.674608 12749 raft_consensus.cc:686] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:24.673228 12764 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.675559 12749 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.677772 12764 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.663275 10187 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:24.657492 12871 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:24.662138 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.678866 12871 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.679033 10264 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:24.679275 12871 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.679679 12783 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.662248 12878 tablet_replica.cc:330] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: stopping tablet replica
I20240507 08:06:24.682695 12878 raft_consensus.cc:2229] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:24.683149 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.683547 12878 raft_consensus.cc:2258] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:24.683856 10336 raft_consensus.cc:2379] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:24.685900 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:24.688131 10155 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:24.689981 10155 leader_election.cc:398] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9.
I20240507 08:06:24.690567 10155 leader_election.cc:401] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:24.691171 12818 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9.
I20240507 08:06:24.693928 10264 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:24.708406 12860 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:24.706719 10341 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:24.713055 10341 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:24.721029 10183 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:24.697260 10185 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:24.717213 10232 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:24.724011 12060 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.725451 12764 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:24.731745 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.723143 12794 raft_consensus.cc:2741] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20240507 08:06:24.732707 12704 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:24.733379 12704 raft_consensus.cc:686] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:24.733661 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.734282 12704 consensus_queue.cc:237] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.743392 10306 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.728657 10088 catalog_manager.cc:5484] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } }
I20240507 08:06:24.744984 12060 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.749481 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.750509 10265 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:24.750186 10192 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:24.752117 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.758227 10340 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:24.753449 10232 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:24.768260 10233 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:24.764444 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.766952 10306 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.770350 12060 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:24.771745 10306 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.772199 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:24.772919 10333 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.776582 10254 tablet_service.cc:1514] Processing DeleteTablet for tablet 03d2e621edea41ae97e79e36e4dc0365 with delete_type TABLET_DATA_TOMBSTONED (TS bcf08c8276ba4a9ba646ad0f7090fcd8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42910
I20240507 08:06:24.777731 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:24.778705 10187 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 7: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:24.759598 12780 raft_consensus.cc:2741] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240507 08:06:24.781805 12764 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:24.775728 12872 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.782477 12866 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:24.808573 10306 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.793598 12860 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:24.814219 12860 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:24.822979 12860 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
W20240507 08:06:24.824384 12879 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.209s user 0.003s sys 0.000s
I20240507 08:06:24.826522 12873 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:24.826965 12873 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:24.827354 12873 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.797437 12764 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.831192 12766 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:24.842054 12766 raft_consensus.cc:686] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:24.843017 12766 consensus_queue.cc:237] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.832140 12866 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:24.848417 12875 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:24.851132 12875 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:24.815517 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 105249us
I20240507 08:06:24.849260 10333 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:24.856746 10907 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.864857 12786 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:24.867132 12764 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:24.865335 12786 raft_consensus.cc:686] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:24.846632 12866 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:24.882640 10155 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:24.889081 12878 ts_tablet_manager.cc:1875] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:24.902451 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:24.920190 10187 raft_consensus.cc:2379] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:24.920759 10332 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
W20240507 08:06:24.927266 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 107251us
I20240507 08:06:24.930469 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:24.939898 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 168382us
I20240507 08:06:24.927446 12807 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:24.939740 10306 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:24.941504 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.942130 10265 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 9
I20240507 08:06:24.942126 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.943341 10264 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
W20240507 08:06:24.943867 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 87282.3us
I20240507 08:06:24.948125 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.927949 10187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:24.950392 10187 raft_consensus.cc:595] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:24.948006 12792 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20240507 08:06:24.952996 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.953603 10259 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:24.961539 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.962303 10266 raft_consensus.cc:3037] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:24.952454 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.963194 10258 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:24.966914 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:24.968132 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.968679 10263 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:24.969696 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:24.949836 12870 raft_consensus.cc:2781] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:24.972056 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:24.943338 12783 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20240507 08:06:25.008210 10265 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:25.013245 10233 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:25.054175 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:25.055063 10943 raft_consensus.cc:2354] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:24.967056 12871 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.055012 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.076301 10260 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:25.047253 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:24.946014 12786 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:25.086735 10263 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:25.089097 10233 leader_election.cc:398] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Vote denied by peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849) with higher term. Message: Invalid argument: T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:25.092339 10233 leader_election.cc:401] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20240507 08:06:25.055514 12890 tablet_replica.cc:330] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: stopping tablet replica
I20240507 08:06:25.054227 12729 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:25.101239 12896 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849) with higher term. Message: Invalid argument: T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6.
I20240507 08:06:25.104034 12860 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:25.105123 10258 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:25.114892 10258 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:25.099763 12729 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:25.134219 12729 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:25.136198 12873 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.094461 12764 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:25.098860 12867 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:25.151409 12807 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:25.151862 12807 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:25.153376 10266 raft_consensus.cc:2454] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:25.156039 12871 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:25.157536 12891 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.145s user 0.000s sys 0.000s
I20240507 08:06:25.133405 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:25.158576 10939 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:25.161353 12887 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.340s user 0.001s sys 0.000s
W20240507 08:06:25.163573 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 96268.5us
I20240507 08:06:25.164786 10264 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:25.164697 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.165894 10266 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:25.170337 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.162539 12867 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:25.194806 12867 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:25.227226 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:25.218585 12860 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:25.232093 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:25.242414 10266 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:25.249646 12873 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:25.254374 12903 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240507 08:06:25.257901 10341 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:25.260488 10304 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:25.262926 12869 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:25.263419 12869 raft_consensus.cc:686] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:25.264286 12869 consensus_queue.cc:237] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.264853 12878 ts_tablet_manager.cc:1888] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.8
I20240507 08:06:25.260905 12905 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20240507 08:06:25.262050 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.265861 10260 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:25.270252 10261 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:25.265391 12878 log.cc:1196] T 7a181f4ae4d342129ad262e5545debc9 P 88a26dac37fa4243b8c2b3d47f403488: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-2-root/wals/7a181f4ae4d342129ad262e5545debc9
I20240507 08:06:25.284718 10939 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:25.289424 12867 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.290679 10332 raft_consensus.cc:2354] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
W20240507 08:06:25.281873 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 132864us
I20240507 08:06:25.249356 12729 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.318892 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.321544 10187 raft_consensus.cc:2354] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4.
I20240507 08:06:25.322475 10190 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:25.337728 10187 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:25.344640 10192 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:25.345134 10185 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:25.313721 12870 raft_consensus.cc:686] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:25.347625 10261 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:25.352304 10304 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:25.353156 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.353765 10183 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:25.359815 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.360371 10192 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:25.367512 12910 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20240507 08:06:25.370152 12890 raft_consensus.cc:2229] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240507 08:06:25.370651 12729 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:25.371099 12890 raft_consensus.cc:2258] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240507 08:06:25.373538 10192 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:25.374717 10190 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:25.361709 10158 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:25.378564 10183 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:25.382472 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:25.392969 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:25.394219 10155 leader_election.cc:398] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8.
I20240507 08:06:25.394657 10155 leader_election.cc:401] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:25.375905 12780 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:25.392566 10267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:25.397892 10267 raft_consensus.cc:595] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:25.398473 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.399221 10260 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 7
I20240507 08:06:25.392717 12890 ts_tablet_manager.cc:1875] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240507 08:06:25.395624 10934 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:25.393297 12867 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:25.408847 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 60800.1us
I20240507 08:06:25.430423 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:25.432852 12824 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:25.435395 10306 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:25.452198 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 90608.2us
I20240507 08:06:25.409963 12780 raft_consensus.cc:686] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:25.453634 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56970.6us
I20240507 08:06:25.458000 10341 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20240507 08:06:25.433542 12916 log.cc:924] Time spent T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20240507 08:06:25.460702 10232 leader_election.cc:398] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4.
I20240507 08:06:25.461014 10341 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:25.461701 10232 leader_election.cc:401] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20240507 08:06:25.435609 12892 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:25.470424 12904 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8.
I20240507 08:06:25.472250 10183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:25.472873 10183 raft_consensus.cc:595] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:25.451020 12870 consensus_queue.cc:237] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.437814 12807 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.479285 12807 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:25.479419 12889 raft_consensus.cc:2741] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20240507 08:06:25.481525 10155 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:25.485949 10187 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:25.487023 12780 consensus_queue.cc:237] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.491672 12921 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:25.492245 12921 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:25.496518 12927 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:25.293985 10075 catalog_manager.cc:4830] TS 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047): tablet 7a181f4ae4d342129ad262e5545debc9 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
W20240507 08:06:25.498373 10075 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 205572us
I20240507 08:06:25.504805 12915 raft_consensus.cc:979] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:25.505244 12915 raft_consensus.cc:1067] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:25.498193 10086 catalog_manager.cc:5484] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:25.501922 12906 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:25.507894 10260 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:25.509649 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.510353 10267 raft_consensus.cc:2354] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:25.510564 10190 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:25.511111 10190 raft_consensus.cc:595] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:25.512678 10333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:25.513295 10333 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:25.513702 12913 raft_consensus.cc:979] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:25.513792 10333 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:25.514205 12913 raft_consensus.cc:1067] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:25.515269 12874 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:25.515801 12874 raft_consensus.cc:686] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:25.521375 10934 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20240507 08:06:25.523700 12920 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240507 08:06:25.526429 10155 leader_election.cc:398] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:25.527887 10260 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:25.530282 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.530998 10190 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:25.526825 10155 leader_election.cc:401] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20240507 08:06:25.536870 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.535703 10333 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:25.548018 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 81492.8us
I20240507 08:06:25.547291 10333 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:25.555151 10334 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:25.555763 10334 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:25.556111 10334 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:25.557184 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 102618us
I20240507 08:06:25.559296 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.559834 10183 raft_consensus.cc:3032] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:25.560104 10183 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:25.555099 12892 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:25.562207 10192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:25.562796 10192 raft_consensus.cc:595] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:25.562634 12892 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.565274 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: true dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:25.555298 12873 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20240507 08:06:25.565935 10934 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:25.551843 12889 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20240507 08:06:25.493131 12921 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.569913 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:25.571622 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:25.572309 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:25.573797 12914 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:25.516614 12874 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:25.577948 10183 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:25.578907 10183 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:25.593925 10190 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:25.595211 10306 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:25.592289 10183 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:25.551667 12913 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5.
I20240507 08:06:25.604858 10334 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.605563 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:25.615290 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:25.615882 10183 raft_consensus.cc:3032] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:25.616171 10183 raft_consensus.cc:729] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:25.616791 10183 consensus_queue.cc:260] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.620999 12923 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4.
I20240507 08:06:25.621492 12908 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:25.622007 10333 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:25.622053 12908 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:25.629142 10088 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:25.603760 12764 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240507 08:06:25.656517 10334 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:25.610075 12824 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
W20240507 08:06:25.671573 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 58539.9us
I20240507 08:06:25.676347 10259 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:25.683287 12899 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20240507 08:06:25.684954 12919 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:25.685075 10934 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:25.685736 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:25.686303 10262 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:25.688278 12060 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:25.626293 12783 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240507 08:06:25.691999 12787 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20240507 08:06:25.703727 10262 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:25.707293 10304 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:25.694231 12919 raft_consensus.cc:686] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:25.696260 12704 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20240507 08:06:25.730381 12807 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:25.744839 10231 leader_election.cc:398] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:25.745191 10231 leader_election.cc:401] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20240507 08:06:25.748486 12929 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240507 08:06:25.756994 12824 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:25.773326 10185 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:25.755481 12807 raft_consensus.cc:686] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:25.732442 12908 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:25.875725 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 89085.8us
I20240507 08:06:25.782368 12871 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20240507 08:06:25.922705 12907 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:25.951232 12907 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:25.953410 12907 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:25.955746 12939 log.cc:924] Time spent T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240507 08:06:25.778760 12919 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:25.958295 12937 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
W20240507 08:06:26.012140 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 102597us
I20240507 08:06:26.073709 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:26.076757 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 70163.6us
I20240507 08:06:26.094859 10183 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:25.946288 12807 consensus_queue.cc:237] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:26.079257 12940 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20240507 08:06:26.076045 12890 ts_tablet_manager.cc:1888] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.6
I20240507 08:06:26.124751 10262 raft_consensus.cc:1261] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:26.131364 10266 raft_consensus.cc:1261] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:26.133240 10258 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:26.138813 10183 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:26.138140 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.136703 12890 log.cc:1196] T 03d2e621edea41ae97e79e36e4dc0365 P bcf08c8276ba4a9ba646ad0f7090fcd8: Deleting WAL directory at /tmp/dist-test-task1xXoxz/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1715069050956306-9046-0/minicluster-data/ts-1-root/wals/03d2e621edea41ae97e79e36e4dc0365
W20240507 08:06:26.209476 12945 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20240507 08:06:26.303663 10183 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:26.308688 10333 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
W20240507 08:06:26.419351 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 117102us
W20240507 08:06:26.395452 12869 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s user 0.005s sys 0.005s
W20240507 08:06:26.456799 12869 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s user 0.005s sys 0.006s
W20240507 08:06:26.456853 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 60496.1us
I20240507 08:06:26.280009 12890 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42910 (request call id 495) took 1475 ms. Trace:
W20240507 08:06:26.460453 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 96442.1us
W20240507 08:06:26.505717 12970 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.162s user 0.001s sys 0.001s
W20240507 08:06:26.511489 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 107726us
I20240507 08:06:26.407563 12964 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20240507 08:06:26.514398 10231 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:26.521343 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 112266us
I20240507 08:06:26.517277 12674 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:26.489555 12966 consensus_queue.cc:1035] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20240507 08:06:26.566948 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 57025.5us
I20240507 08:06:26.521178 12871 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20240507 08:06:26.572121 12968 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:26.592247 12968 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:26.590656 12974 log.cc:924] Time spent T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:06:26.604238 12968 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:26.538919 12870 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20240507 08:06:26.464809 12890 rpcz_store.cc:276] 0507 08:06:24.758693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:24.776268 (+ 17575us) service_pool.cc:225] Handling call
0507 08:06:24.862791 (+ 86523us) ts_tablet_manager.cc:1120] Deleting tablet 03d2e621edea41ae97e79e36e4dc0365
0507 08:06:24.862851 (+ 60us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:06:25.573092 (+710241us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:26.004397 (+431305us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:26.220236 (+215839us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":181499,"tablet-delete.queue_time_us":85623,"thread_start_us":78107,"threads_started":1}
I20240507 08:06:26.549415 12907 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:26.610175 12907 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:26.611032 12907 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.611625 12749 raft_consensus.cc:979] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: : Instructing follower 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:26.612123 12749 raft_consensus.cc:1067] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Signalling peer 88a26dac37fa4243b8c2b3d47f403488 to start an election
I20240507 08:06:26.612301 12907 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:26.616238 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:26.617954 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47322.3us
I20240507 08:06:26.548377 12978 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:26.544152 12674 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.659440 10336 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:39714
I20240507 08:06:26.660193 10336 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:26.660583 10336 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
W20240507 08:06:26.659350 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 111297us
I20240507 08:06:26.661897 10187 raft_consensus.cc:1261] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:26.666503 10187 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:26.670902 10074 catalog_manager.cc:4830] TS bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081): tablet 03d2e621edea41ae97e79e36e4dc0365 (table test-workload [id=3a18a0ff68ea4970981f80ec29a3f24f]) successfully deleted
I20240507 08:06:26.673812 10184 raft_consensus.cc:1261] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:26.650589 12968 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:26.685431 10189 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:26.708312 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:26.711364 10183 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:26.696542 12968 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:26.728859 12968 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:26.716862 12978 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:26.731423 12978 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:26.734808 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 189685us
W20240507 08:06:26.759047 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 123032us
I20240507 08:06:26.764627 10934 raft_consensus.cc:1261] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:26.767267 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.764705 12674 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:26.759876 12978 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:26.772501 12866 consensus_queue.cc:1035] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20240507 08:06:26.786979 12984 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20240507 08:06:26.790364 12919 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:26.792351 12919 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:26.742182 12956 consensus_queue.cc:1035] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20240507 08:06:26.795631 12959 consensus_queue.cc:1035] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:26.795940 12956 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:26.816139 12983 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:26.816747 12983 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.818818 10336 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.796736 12956 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.830889 10336 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:26.845160 12988 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:26.845757 12988 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:26.862859 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.874296 12988 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:26.915391 12860 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:26.916074 12860 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.904491 12824 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:26.918503 12824 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.919641 12983 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:26.920550 12824 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:26.927172 12999 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:26.928608 12994 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:26.929155 12994 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:26.931247 12994 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:26.927690 12999 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:26.935518 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.923537 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.951608 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.954928 12889 raft_consensus.cc:979] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: : Instructing follower bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:26.955091 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:26.955502 12889 raft_consensus.cc:1067] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Signalling peer bcf08c8276ba4a9ba646ad0f7090fcd8 to start an election
I20240507 08:06:26.957093 12860 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:26.958782 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:26.965368 12928 raft_consensus.cc:979] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: : Instructing follower 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:26.965834 12928 raft_consensus.cc:1067] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Signalling peer 59382e6647c549d89d83c70233c0667e to start an election
I20240507 08:06:26.967608 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:26.968192 10188 raft_consensus.cc:3032] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:26.968456 10188 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:26.970994 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:26.970199 12999 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:26.970661 10188 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:26.974764 12908 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:27.029740 12944 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:27.030411 12944 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.031579 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:27.032461 12944 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.032872 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:27.120225 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 67385.2us
I20240507 08:06:27.125279 10191 raft_consensus.cc:1261] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:27.125852 12956 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:27.131037 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 132469us
I20240507 08:06:27.159413 12858 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:27.160122 12858 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:27.168396 13005 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:27.168737 12783 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:27.168978 13005 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:27.170821 12938 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 1.253s user 0.005s sys 0.008s
I20240507 08:06:27.171680 13005 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:27.165207 12896 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:27.174399 12983 consensus_queue.cc:1035] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:27.174113 12896 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.177613 12896 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.174950 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:27.169312 12783 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.127573 12907 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:27.034705 12908 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.195714 12908 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:27.185149 12783 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.177523 12704 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:27.209728 12704 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:27.215670 12979 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:27.217873 12979 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.219357 12704 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:27.220726 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:27.221663 10939 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:27.222568 12858 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:27.205955 12907 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.224581 10233 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:27.225109 12858 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:27.225564 12704 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:27.226071 12704 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:27.225994 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:27.228760 12704 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:27.227989 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:27.226950 12979 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.225672 12858 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.231971 12858 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.241803 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:27.245313 10188 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:27.248800 12907 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.255017 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:27.255861 10334 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:27.257686 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:27.258464 10334 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:27.259749 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:27.261368 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 429892us
I20240507 08:06:27.266474 12936 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:27.273133 10267 raft_consensus.cc:1261] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:27.276468 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:27.277261 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 110392us
I20240507 08:06:27.266472 10908 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:27.279249 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:27.273456 10188 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:27.280982 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:27.302901 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:27.303503 10263 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:27.302268 12787 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:27.306362 10261 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20240507 08:06:27.305317 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:27.309237 10265 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20240507 08:06:27.309708 10265 pending_rounds.cc:76] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 4
I20240507 08:06:27.305975 12787 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.309974 10265 pending_rounds.cc:97] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240507 08:06:27.316112 10265 raft_consensus.cc:2866] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:27.310776 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:27.317314 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:27.316874 10304 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:27.323658 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:27.322247 10306 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:27.325027 12921 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:27.325234 12981 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20240507 08:06:27.325042 12912 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:27.326782 12948 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
W20240507 08:06:27.327188 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47084.6us
I20240507 08:06:27.325605 12807 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:27.328591 10304 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:27.325138 12722 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:27.328372 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:27.319314 12858 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:27.335363 12927 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:27.334591 12722 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:27.342736 10265 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:42914
I20240507 08:06:27.343281 10265 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:27.343585 10265 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:27.343665 12787 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:27.343950 12722 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:27.345835 10188 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:27.324441 12873 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:27.345335 10191 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:27.350708 10306 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:27.350797 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:27.352591 10263 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:27.356160 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:27.344463 12858 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:27.282366 12936 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.357755 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:27.347227 12807 raft_consensus.cc:686] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:27.366466 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:27.366772 10906 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.011s sys 0.022s
W20240507 08:06:27.367117 10906 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.568s user 0.011s sys 0.022s
I20240507 08:06:27.367614 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:27.375444 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:27.386255 10332 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.615s user 0.023s sys 0.044s
W20240507 08:06:27.386679 10332 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.616s user 0.023s sys 0.044s
I20240507 08:06:27.390863 10265 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:27.364872 12979 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:27.360658 12858 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:27.396914 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:27.397132 10265 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:27.401268 10233 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:27.402776 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:27.386732 12703 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.014s sys 0.033s
I20240507 08:06:27.375568 12704 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:27.405720 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:27.434684 10267 raft_consensus.cc:1226] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:27.406767 12926 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:27.407611 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:27.388690 12783 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:27.430311 10940 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240507 08:06:27.256449 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:28.080170 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:27.854939 10088 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 122) took 2350 ms. Trace:
W20240507 08:06:28.079968 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 679070us
I20240507 08:06:27.611377 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:27.678686 10907 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.012s sys 0.031s
W20240507 08:06:27.608698 10190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.759s user 0.009s sys 0.030s
W20240507 08:06:28.083914 10190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.234s user 0.015s sys 0.042s
W20240507 08:06:28.081454 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 679459us
I20240507 08:06:27.434913 10336 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:27.791885 10189 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.011s sys 0.031s
W20240507 08:06:28.086748 10189 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.236s user 0.014s sys 0.039s
W20240507 08:06:27.621662 10192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.013s sys 0.027s
W20240507 08:06:28.087548 10192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.245s user 0.030s sys 0.028s
W20240507 08:06:28.083382 10907 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.314s user 0.017s sys 0.040s
I20240507 08:06:27.396812 12977 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:27.849043 10086 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59936 (request call id 215) took 2880 ms. Trace:
W20240507 08:06:27.635023 10185 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s user 0.024s sys 0.057s
W20240507 08:06:28.095786 10185 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.245s user 0.032s sys 0.076s
I20240507 08:06:27.723150 12858 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:27.394800 12896 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:27.759686 10156 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.016s user 0.025s sys 0.026s
W20240507 08:06:28.099846 10156 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.358s user 0.030s sys 0.033s
W20240507 08:06:27.596527 10307 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.018s sys 0.026s
W20240507 08:06:28.100766 10307 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.332s user 0.025s sys 0.039s
W20240507 08:06:28.101094 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 1.333e+06us
I20240507 08:06:28.098569 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1678) took 1127 ms. Trace:
W20240507 08:06:28.100216 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 1.37563e+06us
I20240507 08:06:28.101605 12704 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.107795 12921 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:28.108319 12921 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:27.434988 12824 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:28.111949 10266 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:28.030558 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:28.090905 12926 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:28.114022 12926 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:28.107837 12943 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.115998 12943 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.090142 12783 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.097477 12858 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.117923 12783 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:28.119313 12951 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.119441 12858 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.119819 12951 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.103700 10191 rpcz_store.cc:276] 0507 08:06:26.971131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.425365 (+454234us) service_pool.cc:225] Handling call
0507 08:06:28.097993 (+672628us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.098196 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.098229 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.098278 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:28.098320 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.098395 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.098508 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:28.082437 10088 rpcz_store.cc:276] 0507 08:06:25.504457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:25.543273 (+ 38816us) service_pool.cc:225] Handling call
0507 08:06:27.854860 (+2311587us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:26.305288 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:26.305831 (+ 543us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:26.305870 (+ 39us) write_op.cc:435] Acquired schema lock
0507 08:06:26.305884 (+ 14us) tablet.cc:660] Decoding operations
0507 08:06:26.306216 (+ 332us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:26.306266 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:26.306295 (+ 29us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:06:26.306475 (+ 180us) tablet.cc:699] Row locks acquired
0507 08:06:26.306491 (+ 16us) write_op.cc:260] PREPARE: finished
0507 08:06:26.306625 (+ 134us) write_op.cc:270] Start()
0507 08:06:26.306735 (+ 110us) write_op.cc:276] Timestamp: P: 1715069186306579 usec, L: 0
0507 08:06:26.306756 (+ 21us) op_driver.cc:331] REPLICATION: starting
0507 08:06:26.406250 (+ 99494us) log.cc:842] Serialized 536 byte log entry
0507 08:06:26.689851 (+283601us) op_driver.cc:444] REPLICATION: finished
0507 08:06:26.783657 (+ 93806us) write_op.cc:301] APPLY: starting
0507 08:06:26.783726 (+ 69us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:26.783820 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:26.783922 (+ 102us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:26.784157 (+ 235us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:26.784303 (+ 146us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:06:26.784351 (+ 48us) write_op.cc:312] APPLY: finished
0507 08:06:26.784510 (+ 159us) log.cc:842] Serialized 23 byte log entry
0507 08:06:26.784672 (+ 162us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:26.784742 (+ 70us) write_op.cc:454] Released schema lock
0507 08:06:26.784777 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:26.794770 (+ 9993us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:26.797931 (+ 3161us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":908278,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":93590,"apply.run_cpu_time_us":1634,"apply.run_wall_time_us":14485,"num_ops":1,"prepare.queue_time_us":517232,"prepare.run_cpu_time_us":2902,"prepare.run_wall_time_us":104201,"replication_time_us":382981,"thread_start_us":245973,"threads_started":1}]]}
W20240507 08:06:28.038293 12838 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.011s sys 0.020s
I20240507 08:06:28.143469 10908 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:28.174077 10266 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:28.129321 10089 catalog_manager.cc:5484] T a34da9500ed541909dbdbcc3be70a067 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 7, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
W20240507 08:06:27.430459 12703 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s user 0.014s sys 0.033s
I20240507 08:06:28.193514 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:28.187456 10336 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:28.207381 10265 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:27.935500 12986 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:28.099491 12977 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:28.176263 12838 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.482s user 0.018s sys 0.023s
W20240507 08:06:28.204355 10187 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.246s user 0.013s sys 0.040s
I20240507 08:06:27.659166 12807 consensus_queue.cc:237] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.212548 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:28.503528 10942 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:28.503595 10906 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:28.389914 12722 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:28.434507 12949 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.507416 10907 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:28.502591 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2065) took 1235 ms. Trace:
I20240507 08:06:28.123225 12978 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:28.508484 10191 rpcz_store.cc:276] 0507 08:06:27.267309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.165362 (+898053us) service_pool.cc:225] Handling call
0507 08:06:28.165727 (+ 365us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.165910 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.165937 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.165969 (+ 32us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:28.165999 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.166049 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.502530 (+336481us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:28.508869 12978 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.510669 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2067) took 1099 ms. Trace:
I20240507 08:06:28.193015 12998 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.513114 12998 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.389560 12951 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:28.515451 12951 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.516083 12951 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.137868 12966 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.519225 12966 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.233358 12869 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.520848 12869 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.395457 12943 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.416646 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 419) took 1047 ms. Trace:
I20240507 08:06:28.525466 10261 rpcz_store.cc:276] 0507 08:06:27.369643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.237732 (+868089us) service_pool.cc:225] Handling call
0507 08:06:28.415963 (+178231us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.416242 (+ 279us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.416279 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.416328 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:28.416371 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.416441 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.416575 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:28.102661 12936 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:28.527673 10187 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.569s user 0.015s sys 0.041s
I20240507 08:06:28.206403 12912 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:28.529379 12912 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:28.309322 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 205314us
I20240507 08:06:28.138072 12959 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:28.090548 10086 rpcz_store.cc:276] 0507 08:06:24.933854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:24.934041 (+ 187us) service_pool.cc:225] Handling call
0507 08:06:27.814015 (+2879974us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:25.669904 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:25.753840 (+ 83936us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:25.755299 (+ 1459us) write_op.cc:435] Acquired schema lock
0507 08:06:25.757260 (+ 1961us) tablet.cc:660] Decoding operations
0507 08:06:25.880474 (+123214us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:25.911586 (+ 31112us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:25.915280 (+ 3694us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:06:25.944991 (+ 29711us) tablet.cc:699] Row locks acquired
0507 08:06:25.945024 (+ 33us) write_op.cc:260] PREPARE: finished
0507 08:06:25.945238 (+ 214us) write_op.cc:270] Start()
0507 08:06:25.945391 (+ 153us) write_op.cc:276] Timestamp: P: 1715069185945173 usec, L: 0
0507 08:06:25.945421 (+ 30us) op_driver.cc:331] REPLICATION: starting
0507 08:06:26.138630 (+193209us) log.cc:842] Serialized 541 byte log entry
0507 08:06:26.532232 (+393602us) op_driver.cc:444] REPLICATION: finished
0507 08:06:26.775789 (+243557us) write_op.cc:301] APPLY: starting
0507 08:06:26.781730 (+ 5941us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:26.781838 (+ 108us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:26.781973 (+ 135us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:26.782194 (+ 221us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:26.782357 (+ 163us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:06:26.782411 (+ 54us) write_op.cc:312] APPLY: finished
0507 08:06:26.782595 (+ 184us) log.cc:842] Serialized 23 byte log entry
0507 08:06:26.782820 (+ 225us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:26.782903 (+ 83us) write_op.cc:454] Released schema lock
0507 08:06:26.782953 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:26.783231 (+ 278us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:26.783265 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":206567,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":216024,"apply.run_cpu_time_us":1935,"apply.run_wall_time_us":20541,"num_ops":1,"prepare.queue_time_us":111431,"prepare.run_cpu_time_us":27316,"prepare.run_wall_time_us":537622,"replication_time_us":586698,"thread_start_us":158936,"threads_started":2}]]}
I20240507 08:06:28.511066 10191 rpcz_store.cc:276] 0507 08:06:27.411239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.509860 (+1098621us) service_pool.cc:225] Handling call
0507 08:06:28.510196 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.510355 (+ 159us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.510382 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.510422 (+ 40us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:28.510455 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.510505 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.510615 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:28.537663 10184 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.565s user 0.014s sys 0.061s
W20240507 08:06:28.543200 10184 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.571s user 0.015s sys 0.061s
I20240507 08:06:28.434896 13001 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:28.535017 12949 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.505693 12722 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.501519 13014 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.308746 12968 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.532395 12977 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:28.135378 12896 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.546370 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 421) took 1176 ms. Trace:
I20240507 08:06:28.212697 12981 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.532166 12885 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:28.579200 12992 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:28.579813 12924 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20240507 08:06:28.577428 10265 rpcz_store.cc:276] 0507 08:06:27.370339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.308527 (+938188us) service_pool.cc:225] Handling call
0507 08:06:28.545754 (+237227us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.545960 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.546000 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.546054 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:28.546097 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.546166 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.546304 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:28.567718 12978 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.557332 12976 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:28.592456 10264 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.284s user 0.020s sys 0.042s
W20240507 08:06:28.598209 10264 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.289s user 0.020s sys 0.042s
W20240507 08:06:28.593353 10259 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.270s user 0.020s sys 0.041s
W20240507 08:06:28.599264 10259 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.276s user 0.021s sys 0.041s
W20240507 08:06:28.593829 10232 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.673s user 0.042s sys 0.046s
I20240507 08:06:28.568478 12921 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.600610 12921 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.601066 12921 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.602945 12921 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.603390 12921 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:28.603802 12921 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.555603 12936 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:28.606158 10232 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.698s user 0.042s sys 0.046s
W20240507 08:06:28.606421 10183 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.650s user 0.028s sys 0.054s
W20240507 08:06:28.611533 10183 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.655s user 0.028s sys 0.054s
W20240507 08:06:28.597705 10258 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.277s user 0.023s sys 0.065s
W20240507 08:06:28.612417 10258 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.292s user 0.024s sys 0.065s
I20240507 08:06:28.588835 12913 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.583700 12980 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.556449 12986 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:28.610252 12786 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.343s user 0.054s sys 0.103s
I20240507 08:06:28.638334 12960 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.639065 12960 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.565219 13001 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.592990 12783 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.569844 12968 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.556912 12959 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.650337 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2047) took 1831 ms. Trace:
I20240507 08:06:28.653941 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2045) took 1841 ms. Trace:
I20240507 08:06:28.658638 10265 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:28.622401 12869 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:28.568898 13014 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.663389 10189 rpcz_store.cc:276] 0507 08:06:26.819328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.819463 (+ 135us) service_pool.cc:225] Handling call
0507 08:06:26.824900 (+ 5437us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.829700 (+ 4800us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.829728 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.830118 (+ 390us) log.cc:842] Serialized 26 byte log entry
0507 08:06:27.678243 (+848125us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b48000600d8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:06:28.087086 (+408843us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:28.087123 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.087182 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:28.649901 (+562719us) raft_consensus.cc:1671] finished
0507 08:06:28.649931 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.650233 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":716565,"raft.queue_time_us":7529,"raft.run_cpu_time_us":21639,"raft.run_wall_time_us":845633,"spinlock_wait_cycles":5896448,"thread_start_us":1236697,"threads_started":1,"wal-append.queue_time_us":1816108}
I20240507 08:06:28.645803 13001 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.667807 10942 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:28.591178 12981 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:28.657369 12786 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.391s user 0.055s sys 0.104s
I20240507 08:06:28.646114 12951 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:28.648423 12958 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.664767 10185 rpcz_store.cc:276] 0507 08:06:26.812342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.812830 (+ 488us) service_pool.cc:225] Handling call
0507 08:06:26.813229 (+ 399us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.820059 (+ 6830us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.820093 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.829667 (+ 9574us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.096206 (+1266539us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:28.096265 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.096348 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:28.653672 (+557324us) raft_consensus.cc:1671] finished
0507 08:06:28.653700 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.653871 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99325,"raft.queue_time_us":6975,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6708,"thread_start_us":1245390,"threads_started":1,"wal-append.queue_time_us":1807064}
I20240507 08:06:28.654407 12898 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:28.641564 12960 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:28.704521 10262 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.373s user 0.012s sys 0.035s
I20240507 08:06:28.704865 10232 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:28.660270 12821 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.654624 12783 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.622997 12936 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.668462 12703 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:28.662209 12921 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.651580 13019 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:28.621822 12992 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.648298 13011 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.594601 12885 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.628687 12976 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.641875 13009 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:28.699110 12892 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.696377 12912 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:28.729655 10186 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.625s user 0.023s sys 0.059s
W20240507 08:06:28.732430 10262 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.404s user 0.013s sys 0.035s
W20240507 08:06:28.796836 10186 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.828s user 0.023s sys 0.059s
I20240507 08:06:28.741802 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 420) took 1371 ms. Trace:
W20240507 08:06:28.758260 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 53516.4us
I20240507 08:06:28.706734 12966 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:28.711329 12704 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:28.646193 12913 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.658310 12980 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:28.799032 10332 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.412s user 0.020s sys 0.018s
I20240507 08:06:28.797317 10267 rpcz_store.cc:276] 0507 08:06:27.370029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.285725 (+915696us) service_pool.cc:225] Handling call
0507 08:06:28.729682 (+443957us) spinlock_profiling.cc:243] Waited 7.37 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:06:28.741119 (+ 11437us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.741347 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.741387 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.741447 (+ 60us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:28.741489 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.741566 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.741730 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16955264}
I20240507 08:06:28.775053 12975 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:28.802059 10089 catalog_manager.cc:5484] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:28.779619 13019 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:28.792760 13009 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
W20240507 08:06:28.807276 10332 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.420s user 0.020s sys 0.018s
I20240507 08:06:28.794942 12949 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:28.783883 12998 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.775575 12821 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.845393 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1577) took 2070 ms. Trace:
I20240507 08:06:28.804064 12912 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:28.805519 12722 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.775736 12703 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:28.822295 12867 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.625s user 0.028s sys 0.045s
I20240507 08:06:28.805119 12896 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.733011 12958 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.810619 13019 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
W20240507 08:06:28.854430 10337 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.681s user 0.043s sys 0.046s
I20240507 08:06:28.789692 13011 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.771332 12898 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:28.799857 12892 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:28.803853 12704 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.340775 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1882) took 1156 ms. Trace:
I20240507 08:06:29.341357 10265 rpcz_store.cc:276] 0507 08:06:28.184531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.595814 (+411283us) service_pool.cc:225] Handling call
0507 08:06:28.622128 (+ 26314us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.672945 (+ 50817us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.340483 (+667538us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b540023c3d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:29.340707 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76790528}
I20240507 08:06:29.343334 10265 raft_consensus.cc:1261] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:28.853874 10332 rpcz_store.cc:276] 0507 08:06:26.741779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.741947 (+ 168us) service_pool.cc:225] Handling call
0507 08:06:26.742222 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:26.742433 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 6
0507 08:06:27.387212 (+644779us) log.cc:842] Serialized 14 byte log entry
0507 08:06:28.811431 (+1424219us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.811497 (+ 66us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:28.811560 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.811641 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:28.811780 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":45825,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":817,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":103,"thread_start_us":2038741,"threads_started":2}
W20240507 08:06:28.873659 12867 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.676s user 0.030s sys 0.045s
W20240507 08:06:29.106930 10260 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.798s user 0.008s sys 0.035s
I20240507 08:06:29.207082 12976 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.927616 12980 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:29.346161 10337 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.350s user 0.047s sys 0.054s
I20240507 08:06:29.346123 12958 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.347491 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1885) took 1154 ms. Trace:
I20240507 08:06:29.349706 12703 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:29.350054 10263 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.990s user 0.025s sys 0.010s
W20240507 08:06:29.383011 10263 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.023s user 0.025s sys 0.011s
W20240507 08:06:29.067585 12926 consensus_meta.cc:287] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.953s user 0.012s sys 0.000s
I20240507 08:06:28.873315 12912 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:29.066499 12959 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:29.352607 12898 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:28.923689 12731 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:29.216919 10231 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.116s user 0.005s sys 0.026s
W20240507 08:06:29.283891 10304 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.190s user 0.028s sys 0.068s
W20240507 08:06:29.418228 10304 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.325s user 0.029s sys 0.068s
W20240507 08:06:28.915494 13015 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.184s user 0.000s sys 0.000s
I20240507 08:06:28.923151 12885 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:28.921907 12968 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:28.867709 12681 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.050s user 0.046s sys 0.068s
I20240507 08:06:29.366281 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1827) took 1160 ms. Trace:
I20240507 08:06:28.867834 12998 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:28.924218 12953 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:29.380792 10260 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.072s user 0.013s sys 0.044s
I20240507 08:06:29.381057 10267 rpcz_store.cc:276] 0507 08:06:28.193431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:29.041240 (+847809us) service_pool.cc:225] Handling call
0507 08:06:29.346852 (+305612us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:29.347065 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:29.347101 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:29.347157 (+ 56us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:29.347213 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.347281 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.347421 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:29.362682 12981 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:29.086762 12986 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:29.251628 12992 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:29.207216 13026 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.338s user 0.000s sys 0.001s
I20240507 08:06:29.277050 12913 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.296489 13009 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:29.418169 10231 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.321s user 0.007s sys 0.028s
I20240507 08:06:29.107040 12978 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:29.428504 12681 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.611s user 0.050s sys 0.080s
I20240507 08:06:29.350350 13014 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.292203 12821 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:29.423255 10188 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.312s user 0.048s sys 0.052s
I20240507 08:06:29.426241 10265 rpcz_store.cc:276] 0507 08:06:28.206126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:29.342558 (+1136432us) service_pool.cc:225] Handling call
0507 08:06:29.342991 (+ 433us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:29.365982 (+ 22991us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.366220 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:29.429904 12944 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:29.434629 13027 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20240507 08:06:29.434927 12968 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:29.349673 13011 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:29.481889 10305 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.975s user 0.018s sys 0.017s
I20240507 08:06:29.455619 12885 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:29.513605 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.493727 10231 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:29.421311 12926 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:29.512307 10305 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.006s user 0.019s sys 0.017s
I20240507 08:06:29.477730 12733 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:29.502058 10908 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.145s user 0.010s sys 0.027s
W20240507 08:06:29.555147 10908 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.198s user 0.011s sys 0.027s
W20240507 08:06:29.555840 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 1.41304e+06us
I20240507 08:06:29.518855 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:29.510187 10188 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.399s user 0.049s sys 0.057s
I20240507 08:06:29.436136 12867 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:29.436574 13025 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.273s user 0.000s sys 0.001s
I20240507 08:06:29.553975 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:29.537948 12681 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:29.423024 12731 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:29.451885 13023 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
W20240507 08:06:29.560868 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 67315.5us
W20240507 08:06:29.561519 10230 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.264s user 0.028s sys 0.066s
I20240507 08:06:29.561560 12913 consensus_queue.cc:1046] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:29.562355 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.479993 12936 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.564571 10155 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:29.477900 12992 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:29.479817 12896 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:29.575464 10332 raft_consensus.cc:2396] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20240507 08:06:29.576678 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:29.580058 10230 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.482s user 0.029s sys 0.066s
I20240507 08:06:29.517374 12986 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:29.579392 10306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.016s user 0.007s sys 0.060s
I20240507 08:06:29.517429 13002 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:29.580770 10907 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.020s user 0.020s sys 0.020s
I20240507 08:06:29.590804 10265 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
W20240507 08:06:29.594795 10306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.032s user 0.007s sys 0.060s
W20240507 08:06:29.595257 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 1.03298e+06us
W20240507 08:06:29.595618 10907 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.046s user 0.020s sys 0.021s
I20240507 08:06:29.573437 12867 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.596930 12979 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:29.597460 12979 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.598039 12867 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:29.598258 12703 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:29.598546 12867 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:29.598803 12703 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.599375 12979 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.599092 12867 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.559115 12892 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:29.602478 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 1.09559e+06us
I20240507 08:06:29.436434 12953 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:29.613655 10187 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.085s user 0.009s sys 0.042s
I20240507 08:06:29.577306 12821 raft_consensus.cc:2781] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
W20240507 08:06:29.662346 10266 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.029s sys 0.057s
W20240507 08:06:29.662623 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 60178.7us
W20240507 08:06:29.664558 10187 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.136s user 0.011s sys 0.049s
I20240507 08:06:29.451697 12978 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:29.517794 12944 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.578502 13011 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:29.551290 12885 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.689828 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:29.690093 10158 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.117s user 0.018s sys 0.036s
I20240507 08:06:29.579376 12704 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.575860 12959 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:29.664294 10157 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.078s user 0.017s sys 0.036s
I20240507 08:06:29.559720 12733 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:29.687036 10266 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.115s user 0.029s sys 0.057s
I20240507 08:06:29.690538 10267 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:29.594466 12976 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:29.580549 12981 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:29.691277 10158 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.118s user 0.018s sys 0.036s
I20240507 08:06:29.606233 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.602835 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1578) took 2851 ms. Trace:
I20240507 08:06:29.686273 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2052) took 2700 ms. Trace:
W20240507 08:06:29.683284 10906 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.138s user 0.021s sys 0.070s
W20240507 08:06:29.583709 13033 log.cc:924] Time spent T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.363s user 0.000s sys 0.001s
I20240507 08:06:29.582026 12681 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:29.704525 10157 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.118s user 0.018s sys 0.041s
I20240507 08:06:29.600788 12896 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.709358 13011 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:29.573333 12968 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:29.758543 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1675) took 2811 ms. Trace:
I20240507 08:06:29.784255 12968 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:29.785319 13017 log.cc:924] Time spent T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 1.050s user 0.007s sys 0.010s
I20240507 08:06:29.786343 12968 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.786567 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2048) took 2965 ms. Trace:
I20240507 08:06:29.747987 12926 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:29.785595 13011 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.752660 12731 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.784215 10183 rpcz_store.cc:276] 0507 08:06:26.946713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.952758 (+ 6045us) service_pool.cc:225] Handling call
0507 08:06:26.953034 (+ 276us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.953356 (+ 322us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.953381 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.956222 (+ 2841us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.612052 (+1655830us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:28.612104 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.612182 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.758268 (+1146086us) raft_consensus.cc:1671] finished
0507 08:06:29.758301 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.758472 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":113,"spinlock_wait_cycles":16128,"thread_start_us":1655384,"threads_started":1,"wal-append.queue_time_us":2156281}
W20240507 08:06:29.786975 10909 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.128s user 0.011s sys 0.027s
I20240507 08:06:29.702428 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2053) took 2736 ms. Trace:
W20240507 08:06:29.787645 10909 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.397s user 0.011s sys 0.028s
I20240507 08:06:29.787994 12733 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.787762 10186 rpcz_store.cc:276] 0507 08:06:26.966251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.967508 (+ 1257us) service_pool.cc:225] Handling call
0507 08:06:26.967853 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.968044 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.968068 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.968531 (+ 463us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.797309 (+1828778us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:28.797363 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.797448 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.702165 (+904717us) raft_consensus.cc:1671] finished
0507 08:06:29.702194 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.702354 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1828401,"threads_started":1,"wal-append.queue_time_us":2131818}
I20240507 08:06:29.789841 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:29.790151 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:29.790827 10233 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:29.787309 12926 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:29.791752 12681 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:29.603596 13032 log.cc:924] Time spent T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20240507 08:06:29.791674 12926 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.593555 12980 consensus_queue.cc:1046] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.116s
I20240507 08:06:29.794080 10087 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59948 (request call id 218) took 5204 ms. Trace:
I20240507 08:06:29.793785 10265 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:29.796033 10087 rpcz_store.cc:276] 0507 08:06:24.569163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:24.590585 (+ 21422us) service_pool.cc:225] Handling call
0507 08:06:29.773447 (+5182862us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":472269,"reported_tablets":6}
I20240507 08:06:29.772683 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 310) took 2360 ms. Trace:
I20240507 08:06:29.662638 12913 raft_consensus.cc:3032] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:29.798198 10191 rpcz_store.cc:276] 0507 08:06:27.412100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.545765 (+1133665us) service_pool.cc:225] Handling call
0507 08:06:28.566783 (+ 21018us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:28.797759 (+230976us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.797791 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:28.797832 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:28.797865 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.772409 (+974544us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b540026c854. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:29.772466 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.772618 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":531035904}
I20240507 08:06:29.799059 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.800093 10262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1813) took 2523 ms. Trace:
W20240507 08:06:29.794236 10906 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.249s user 0.023s sys 0.076s
I20240507 08:06:29.796506 10190 rpcz_store.cc:276] 0507 08:06:26.820797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.820952 (+ 155us) service_pool.cc:225] Handling call
0507 08:06:26.821336 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.828651 (+ 7315us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.828690 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.849705 (+ 21015us) log.cc:842] Serialized 26 byte log entry
0507 08:06:27.404320 (+554615us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b48000600d8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:06:28.084343 (+680023us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:28.084380 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.084443 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.784745 (+1700302us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:29.786382 (+ 1637us) raft_consensus.cc:1671] finished
0507 08:06:29.786402 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.786516 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":697876,"raft.queue_time_us":7267,"raft.run_cpu_time_us":26393,"raft.run_wall_time_us":574865,"spinlock_wait_cycles":306229504,"thread_start_us":1234217,"threads_started":1,"wal-append.queue_time_us":1819395}
I20240507 08:06:29.595131 13009 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:29.802949 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 1.29809e+06us
I20240507 08:06:29.800719 10262 rpcz_store.cc:276] 0507 08:06:27.271041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.271163 (+ 122us) service_pool.cc:225] Handling call
0507 08:06:27.271439 (+ 276us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:27.302416 (+ 30977us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:27.302444 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:27.311066 (+ 8622us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.797049 (+1485983us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:28.797090 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.797157 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.794314 (+997157us) raft_consensus.cc:1671] finished
0507 08:06:29.794335 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.794448 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11938,"raft.queue_time_us":44253,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":3025,"thread_start_us":1468628,"threads_started":1,"wal-append.queue_time_us":2098259}
I20240507 08:06:29.766619 10089 catalog_manager.cc:5484] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:29.812361 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.811852 10089 catalog_manager.cc:5484] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:29.813982 10265 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:29.814687 10089 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:29.815287 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.816927 10265 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:29.816318 10089 catalog_manager.cc:5484] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 3 to 5, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 5 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:29.711594 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.686312 12821 raft_consensus.cc:686] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:29.793231 10184 rpcz_store.cc:276] 0507 08:06:26.965930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.969131 (+ 3201us) service_pool.cc:225] Handling call
0507 08:06:26.969492 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.970204 (+ 712us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.970238 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.972378 (+ 2140us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.543651 (+1571273us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:28.543708 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.543799 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.665779 (+1121980us) raft_consensus.cc:1671] finished
0507 08:06:29.665806 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.665967 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1059,"raft.queue_time_us":3120,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9976,"thread_start_us":1570895,"threads_started":1,"wal-append.queue_time_us":1817124}
I20240507 08:06:29.797416 10337 rpcz_store.cc:276] 0507 08:06:26.744481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.744660 (+ 179us) service_pool.cc:225] Handling call
0507 08:06:26.762430 (+ 17770us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.840303 (+ 77873us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.840338 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.937250 (+ 96912us) log.cc:842] Serialized 26 byte log entry
0507 08:06:29.383988 (+2446738us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:29.384106 (+ 118us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.384220 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.595379 (+211159us) raft_consensus.cc:1671] finished
0507 08:06:29.595400 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.595504 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69930,"raft.queue_time_us":79432,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7780,"thread_start_us":2387879,"threads_started":1,"wal-append.queue_time_us":2263354}
I20240507 08:06:29.830679 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:29.831593 10156 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.217s user 0.038s sys 0.057s
W20240507 08:06:29.831938 10156 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.219s user 0.040s sys 0.059s
W20240507 08:06:29.750049 13016 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.966s user 0.007s sys 0.010s
W20240507 08:06:29.813367 10307 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.309s user 0.013s sys 0.056s
W20240507 08:06:29.858299 10307 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.354s user 0.014s sys 0.056s
I20240507 08:06:29.859123 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.859822 10191 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:29.860430 10089 catalog_manager.cc:5484] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:29.753809 12912 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:29.800644 12943 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
W20240507 08:06:29.811254 13024 log.cc:924] Time spent T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.957s user 0.001s sys 0.017s
I20240507 08:06:29.601796 12992 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.693483 13002 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.931296 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1810) took 2661 ms. Trace:
I20240507 08:06:29.931726 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.932469 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.932633 10184 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:29.933017 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.924535 10942 raft_consensus.cc:1226] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20240507 08:06:29.934100 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.831869 13014 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:29.929200 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:29.934341 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:29.935952 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:29.943176 10261 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.160s user 0.010s sys 0.034s
I20240507 08:06:29.946362 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:29.946933 10261 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.164s user 0.010s sys 0.034s
I20240507 08:06:29.947554 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:29.948877 10908 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:29.946537 10940 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:29.951844 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.931661 10258 rpcz_store.cc:276] 0507 08:06:27.270005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.278917 (+ 8912us) service_pool.cc:225] Handling call
0507 08:06:27.279220 (+ 303us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:27.310933 (+ 31713us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:27.310960 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:27.311319 (+ 359us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.612854 (+1301535us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:28.612908 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.613003 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.931107 (+1318104us) raft_consensus.cc:1671] finished
0507 08:06:29.931128 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:29.931244 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37759,"raft.queue_time_us":37972,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3990,"thread_start_us":1292389,"threads_started":1,"wal-append.queue_time_us":1844154}
I20240507 08:06:29.955816 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.711194 12959 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.953161 10265 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
W20240507 08:06:29.887945 10264 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.289s user 0.030s sys 0.029s
I20240507 08:06:29.604326 12986 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.972117 10190 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
W20240507 08:06:29.929261 13034 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.258s user 0.000s sys 0.000s
I20240507 08:06:29.886206 12704 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:29.831676 12733 consensus_queue.cc:1046] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
W20240507 08:06:29.965266 10264 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.366s user 0.031s sys 0.029s
I20240507 08:06:29.977972 10908 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
W20240507 08:06:29.943267 12786 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.013s sys 0.028s
I20240507 08:06:29.959347 10185 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:29.829501 13009 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.824383 12981 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:29.985157 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.980976 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:29.800885 12913 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:29.930848 12958 raft_consensus.cc:3032] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:29.993690 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:29.935745 12943 raft_consensus.cc:686] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:29.993528 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:30.009832 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1814) took 2738 ms. Trace:
I20240507 08:06:29.958361 12898 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:29.959448 12703 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.008765 12783 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.041852 10264 rpcz_store.cc:276] 0507 08:06:27.271382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.273567 (+ 2185us) service_pool.cc:225] Handling call
0507 08:06:27.273952 (+ 385us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:27.274194 (+ 242us) raft_consensus.cc:1520] Early marking committed up to index 6
0507 08:06:28.598670 (+1324476us) log.cc:842] Serialized 14 byte log entry
0507 08:06:30.007664 (+1408994us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:30.007886 (+ 222us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:30.008048 (+ 162us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.009510 (+ 1462us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.009749 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1811831,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":1207,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"thread_start_us":2684008,"threads_started":2}
I20240507 08:06:29.981794 13011 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:30.034363 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:29.979980 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:29.986654 12807 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.062s user 0.029s sys 0.029s
I20240507 08:06:30.036083 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:30.012655 12786 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.233s user 0.014s sys 0.028s
W20240507 08:06:30.039952 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 62110us
I20240507 08:06:30.050609 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2046) took 3231 ms. Trace:
I20240507 08:06:30.051736 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:30.009469 12926 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.029975 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:29.960559 13014 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.030956 12968 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20240507 08:06:30.056069 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:30.057855 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:30.060138 12943 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.005882 12704 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.034982 12821 consensus_queue.cc:237] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.061688 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.068086 12981 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.231602 12681 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.231786 12981 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.232252 12981 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.228477 12892 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:30.233314 12892 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.233944 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.234445 12733 raft_consensus.cc:3032] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:30.234966 12733 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:30.066659 12913 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.235486 12892 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:30.235993 12913 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:30.232220 12681 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.038668 12978 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:30.186013 10155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.012s sys 0.025s
W20240507 08:06:30.254596 10155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.012s sys 0.029s
W20240507 08:06:30.254961 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 589631us
I20240507 08:06:30.255548 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:30.256006 10155 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:30.157449 10304 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.008s sys 0.022s
W20240507 08:06:30.257069 10304 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.695s user 0.011s sys 0.023s
I20240507 08:06:30.041051 12958 raft_consensus.cc:729] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:30.257992 10304 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:30.038926 12953 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:30.043004 12977 consensus_meta.cc:287] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 1.451s user 0.002s sys 0.011s
I20240507 08:06:30.238910 10265 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
W20240507 08:06:30.187057 10263 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.007s sys 0.020s
W20240507 08:06:30.211758 10232 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.505s (SENT)
I20240507 08:06:30.216207 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1824) took 2822 ms. Trace:
W20240507 08:06:30.273861 10263 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s user 0.010s sys 0.021s
I20240507 08:06:30.274678 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1876) took 2933 ms. Trace:
I20240507 08:06:30.274015 10266 rpcz_store.cc:276] 0507 08:06:27.393891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.379097 (+985206us) service_pool.cc:225] Handling call
0507 08:06:28.379422 (+ 325us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:28.535866 (+156444us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.535902 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:28.543132 (+ 7230us) log.cc:842] Serialized 26 byte log entry
0507 08:06:29.817393 (+1274261us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:29.817431 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.817492 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:30.215949 (+398457us) raft_consensus.cc:1671] finished
0507 08:06:30.215968 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.216117 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":117341,"raft.queue_time_us":122555,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":8005,"thread_start_us":1137511,"threads_started":1,"wal-append.queue_time_us":1649195}
I20240507 08:06:30.275192 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.275133 10263 rpcz_store.cc:276] 0507 08:06:27.341399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.359403 (+ 18004us) service_pool.cc:225] Handling call
0507 08:06:27.359680 (+ 277us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:27.359852 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 7
0507 08:06:29.383560 (+2023708us) log.cc:842] Serialized 14 byte log entry
0507 08:06:30.274333 (+890773us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:30.274393 (+ 60us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:30.274440 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.274513 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.274630 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":210,"prepare.queue_time_us":1795864,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":1222,"thread_start_us":2913411,"threads_started":2}
I20240507 08:06:30.237521 10340 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:30.277155 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.278800 12704 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:30.279356 12975 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:30.279568 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.279850 12975 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.280208 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.069324 12783 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.281770 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.284451 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.285482 13043 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240507 08:06:30.286690 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 57063.9us
I20240507 08:06:30.287324 10908 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:30.287667 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.287775 13043 consensus_queue.cc:1035] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240507 08:06:30.288573 13003 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:30.288754 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.289006 13003 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.289398 13003 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:30.289853 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:30.291122 13039 log.cc:924] Time spent T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.226s user 0.000s sys 0.001s
I20240507 08:06:30.292151 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2060) took 3111 ms. Trace:
I20240507 08:06:30.292470 10188 rpcz_store.cc:276] 0507 08:06:27.180872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.109414 (+928542us) service_pool.cc:225] Handling call
0507 08:06:28.109769 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:28.110090 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.110115 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:28.110559 (+ 444us) log.cc:842] Serialized 26 byte log entry
0507 08:06:29.557925 (+1447366us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:29.557961 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.558023 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:30.291961 (+733938us) raft_consensus.cc:1671] finished
0507 08:06:30.291987 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.292100 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":171,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"thread_start_us":1447042,"threads_started":1,"wal-append.queue_time_us":1876475}
I20240507 08:06:30.293761 13036 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:30.294520 10906 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:30.294337 13036 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.296775 13046 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:30.296906 13036 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.209802 10192 rpcz_store.cc:276] 0507 08:06:26.819050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.819181 (+ 131us) service_pool.cc:225] Handling call
0507 08:06:26.819561 (+ 380us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.825791 (+ 6230us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.825834 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.835551 (+ 9717us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.087862 (+1252311us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:28.087896 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:28.087960 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:29.749451 (+1661491us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:30.034867 (+285416us) raft_consensus.cc:1671] finished
0507 08:06:30.034896 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.050526 (+ 15630us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89946,"raft.queue_time_us":8650,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":4961,"spinlock_wait_cycles":242494208,"thread_start_us":1244821,"threads_started":1,"wal-append.queue_time_us":1859893}
W20240507 08:06:30.166265 10259 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.566s user 0.022s sys 0.030s
W20240507 08:06:30.298164 10259 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.698s user 0.023s sys 0.034s
I20240507 08:06:30.298082 10907 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:30.299633 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:30.300727 13046 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:30.303372 12783 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.096740 12885 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.306867 10337 raft_consensus.cc:1226] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:30.271595 12731 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.308269 12731 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.308869 13014 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.309378 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.310245 12731 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.308288 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.312352 10333 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:30.107787 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.238745 10267 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
W20240507 08:06:30.313472 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 57603.4us
W20240507 08:06:30.237700 12838 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.007s sys 0.016s
I20240507 08:06:30.229542 10258 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
W20240507 08:06:30.249572 10306 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.014s sys 0.036s
W20240507 08:06:30.196501 13035 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.155s user 0.000s sys 0.000s
I20240507 08:06:30.265574 10262 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1"
dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
mode: GRACEFUL
new_leader_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:41140
I20240507 08:06:30.134784 12896 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:30.192293 13028 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.165s user 0.000s sys 0.000s
I20240507 08:06:30.048470 12867 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.197363 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.061619 12898 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.157058 12944 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:30.249395 12807 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.313s user 0.035s sys 0.043s
I20240507 08:06:30.273272 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.280177 10940 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:30.288239 10341 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:30.061388 13041 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240507 08:06:30.192829 13022 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
W20240507 08:06:30.314764 10306 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.015s sys 0.036s
I20240507 08:06:30.471689 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.476981 10304 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:30.465620 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 165359us
W20240507 08:06:30.478597 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 877827us
W20240507 08:06:30.476562 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 264957us
I20240507 08:06:30.239272 13011 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.234397 12981 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.051950 12959 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.235744 12733 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.479096 10907 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:30.396803 10187 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.006s sys 0.017s
W20240507 08:06:30.497228 10187 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.805s user 0.010s sys 0.017s
I20240507 08:06:30.281162 13009 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.488394 10334 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:30.318531 10262 raft_consensus.cc:595] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 59382e6647c549d89d83c70233c0667e
I20240507 08:06:30.483050 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1830) took 1048 ms. Trace:
I20240507 08:06:30.499173 10267 rpcz_store.cc:276] 0507 08:06:29.434949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.332555 (+897606us) service_pool.cc:225] Handling call
0507 08:06:30.482627 (+150072us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:30.482767 (+ 140us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:30.482790 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:30.482824 (+ 34us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:30.482850 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.482898 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.482987 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41856}
I20240507 08:06:30.315842 10264 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:30.500697 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.323562 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1812) took 3053 ms. Trace:
I20240507 08:06:30.381500 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1826) took 2981 ms. Trace:
I20240507 08:06:30.366816 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1808) took 3097 ms. Trace:
I20240507 08:06:30.471594 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.478269 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:30.264158 13002 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.263567 12892 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:30.501102 10262 raft_consensus.cc:603] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:30.268847 12986 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.498240 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1674) took 3551 ms. Trace:
I20240507 08:06:30.500303 12952 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.507678 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.509677 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.457818 12978 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:30.394438 13045 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:30.272136 12992 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.495383 10258 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:30.505424 10260 rpcz_store.cc:276] 0507 08:06:27.270677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.270824 (+ 147us) service_pool.cc:225] Handling call
0507 08:06:27.271135 (+ 311us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:27.304002 (+ 32867us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:27.304046 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:27.306336 (+ 2290us) log.cc:842] Serialized 26 byte log entry
0507 08:06:29.428709 (+2122373us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:29.428752 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.428836 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:30.323373 (+894537us) raft_consensus.cc:1671] finished
0507 08:06:30.323400 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.323508 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2918,"raft.queue_time_us":31229,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":9192,"thread_start_us":2120034,"threads_started":1,"wal-append.queue_time_us":2625437}
I20240507 08:06:30.511847 12952 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:30.512709 13056 log.cc:924] Time spent T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.181s user 0.000s sys 0.000s
I20240507 08:06:30.510425 12892 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.323343 13053 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:30.313133 13049 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.502580 10261 rpcz_store.cc:276] 0507 08:06:27.400583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:28.526861 (+1126278us) service_pool.cc:225] Handling call
0507 08:06:28.781710 (+254849us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:06:28.781967 (+ 257us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:28.782436 (+ 469us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:28.782477 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:28.783008 (+ 531us) log.cc:842] Serialized 26 byte log entry
0507 08:06:29.947299 (+1164291us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:29.983039 (+ 35740us) log.cc:842] Serialized 14 byte log entry
0507 08:06:29.983249 (+ 210us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:29.983334 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:30.291294 (+307960us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b18003bf980. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd1b8 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:06:30.381229 (+ 89935us) raft_consensus.cc:1671] finished
0507 08:06:30.381259 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.381431 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52475,"prepare.queue_time_us":39208,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":3765,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":272,"spinlock_wait_cycles":17389696,"thread_start_us":1163894,"threads_started":1,"wal-append.queue_time_us":1446299}
I20240507 08:06:30.506767 10187 rpcz_store.cc:276] 0507 08:06:26.946435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:26.946584 (+ 149us) service_pool.cc:225] Handling call
0507 08:06:26.946999 (+ 415us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:26.948972 (+ 1973us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:26.949014 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:26.958023 (+ 9009us) log.cc:842] Serialized 26 byte log entry
0507 08:06:28.528186 (+1570163us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:29.685874 (+1157688us) log.cc:842] Serialized 14 byte log entry
0507 08:06:29.753737 (+ 67863us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:30.284930 (+531193us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b48000600d8. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf95ccf431 00007fcf95cc6527 00007fcf95cc63ba 00007fcf95dba56f 00007fcf95dba237 00007fcf95dce73c 00007fcf95dce6e8 00007fcf95dce668 00007fcf95dce623 00007fcf95dcd8cc 00007fcf94d509c6
0507 08:06:30.497796 (+212866us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.497914 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:30.497970 (+ 56us) raft_consensus.cc:1671] finished
0507 08:06:30.497986 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.498186 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40485,"prepare.queue_time_us":1505069,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":205583,"raft.run_cpu_time_us":3575,"raft.run_wall_time_us":390532,"spinlock_wait_cycles":95117312,"thread_start_us":3513062,"threads_started":3,"wal-append.queue_time_us":1826533}
I20240507 08:06:30.382125 13047 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:30.322702 12976 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:30.501689 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:30.518303 10907 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:30.497172 13011 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:30.380662 12907 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:30.480274 13041 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:30.317988 12838 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.626s user 0.008s sys 0.016s
I20240507 08:06:30.469991 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.497006 12681 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.497778 13061 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240507 08:06:30.533931 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 55753.4us
I20240507 08:06:30.508158 13042 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:30.273388 12977 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.539908 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.650887 13047 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:30.657207 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 155711us
I20240507 08:06:30.508164 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.517446 13064 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.665335 10191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
from {username='slave'} at 127.0.0.1:55816
I20240507 08:06:30.695583 10191 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240507 08:06:30.696084 10191 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:30.530515 12980 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.624852 13049 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:30.535986 13002 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.526604 12978 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.668988 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:30.535487 13045 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.670282 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.671240 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.675616 12892 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:30.666239 12979 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:30.655820 13041 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:30.660305 13062 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240507 08:06:30.678675 13052 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20240507 08:06:30.663516 13014 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240507 08:06:30.681617 13019 consensus_meta.cc:287] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 1.320s user 0.009s sys 0.002s
I20240507 08:06:30.666133 13047 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:30.685184 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1829) took 1250 ms. Trace:
W20240507 08:06:30.747045 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 82108.3us
I20240507 08:06:30.747948 10265 rpcz_store.cc:276] 0507 08:06:29.434706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.278681 (+843975us) service_pool.cc:225] Handling call
0507 08:06:30.278970 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:30.313232 (+ 34262us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:30.313262 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:30.313694 (+ 432us) log.cc:842] Serialized 26 byte log entry
0507 08:06:30.499990 (+186296us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:30.500027 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.500088 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:30.684922 (+184834us) raft_consensus.cc:1671] finished
0507 08:06:30.684945 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.685098 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26143,"raft.queue_time_us":39257,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":44678,"thread_start_us":186044,"threads_started":1,"wal-append.queue_time_us":349567}
I20240507 08:06:30.689059 10306 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:30.750530 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 61592.2us
I20240507 08:06:30.692114 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.762122 10190 raft_consensus.cc:2396] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240507 08:06:30.721750 12980 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.763371 13050 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:30.693013 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.763864 10191 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.764596 12980 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.539538 12959 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:30.765095 12959 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:30.765465 12959 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:30.499212 12958 consensus_queue.cc:260] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.668525 12898 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.713001 10266 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:30.655582 13061 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:30.777241 10191 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:30.714499 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.656245 12976 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:30.715931 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:30.782284 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 66664.7us
I20240507 08:06:30.783079 10155 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:30.717192 10184 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240507 08:06:30.660610 13042 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.717043 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:30.712929 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 194790us
I20240507 08:06:30.724289 13049 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.708771 13064 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.725921 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.726534 13045 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:30.846359 10186 raft_consensus.cc:2396] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240507 08:06:30.727293 13003 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.734238 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1889) took 1301 ms. Trace:
I20240507 08:06:30.851579 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 1997) took 1270 ms. Trace:
I20240507 08:06:30.851523 10263 rpcz_store.cc:276] 0507 08:06:29.432647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.277311 (+844664us) service_pool.cc:225] Handling call
0507 08:06:30.314090 (+ 36779us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:30.314858 (+ 768us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:30.314895 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:30.314948 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:30.314993 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.315065 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.734064 (+418999us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b54002acbd0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:30.734186 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84009344}
I20240507 08:06:30.503574 10259 rpcz_store.cc:276] 0507 08:06:27.269192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.279524 (+ 10332us) service_pool.cc:225] Handling call
0507 08:06:27.302411 (+ 22887us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:27.302596 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 7
0507 08:06:28.599703 (+1297107us) log.cc:842] Serialized 14 byte log entry
0507 08:06:30.366345 (+1766642us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:30.366421 (+ 76us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:30.366484 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.366582 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:30.366741 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":79917,"prepare.queue_time_us":1807974,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":67638,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":37675,"spinlock_wait_cycles":34688,"thread_start_us":2974821,"threads_started":2,"wal-append.queue_time_us":1899321}
I20240507 08:06:30.719920 12821 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:30.746410 13062 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:30.668545 13053 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.729251 13002 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:30.849135 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 99713.2us
I20240507 08:06:30.850260 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.855111 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.839434 12976 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:30.774310 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.852149 10334 rpcz_store.cc:276] 0507 08:06:29.580910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:29.581077 (+ 167us) service_pool.cc:225] Handling call
0507 08:06:29.581425 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:30.498918 (+917493us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:30.851313 (+352395us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b54002c83d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:30.851522 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2086377856}
I20240507 08:06:30.846720 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.729480 12979 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:30.876960 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.971436 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.925833 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:30.998167 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.022742 10191 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240507 08:06:30.839200 13061 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:31.043099 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.945436 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:30.974385 13072 log.cc:924] Time spent T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.231s user 0.000s sys 0.001s
W20240507 08:06:30.972272 13075 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20240507 08:06:31.047513 10156 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:30.763911 13050 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:31.019289 10184 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:30.847720 13042 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:30.839593 12892 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:30.847445 12898 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:30.903400 13049 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:31.061049 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:30.970294 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:31.067104 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.070827 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.944694 10935 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
W20240507 08:06:30.905885 13073 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20240507 08:06:31.071930 10192 raft_consensus.cc:2379] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:31.001418 12944 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.082512 12944 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.084266 12975 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.076682 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.973677 13064 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.078657 10185 raft_consensus.cc:2454] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:31.041005 13065 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.038710 13074 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.082854 13083 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:31.072717 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:30.904445 12980 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.109769 10184 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:31.109901 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.111297 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.090269 10942 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 211) took 1113 ms. Trace:
I20240507 08:06:31.066733 13050 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:30.773487 13019 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.115329 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.115161 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.112277 10942 rpcz_store.cc:276] 0507 08:06:29.977044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:29.977402 (+ 358us) service_pool.cc:225] Handling call
0507 08:06:30.007898 (+ 30496us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:30.270082 (+262184us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:30.270114 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:30.270157 (+ 43us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:30.270185 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:31.089976 (+819791us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b540055cb54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:31.090039 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:31.090202 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":601482112}
I20240507 08:06:30.903470 12821 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:30.921810 13062 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.074853 12898 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.074914 13049 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.115968 10186 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:31.086053 12977 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.127408 10943 raft_consensus.cc:1226] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:31.125278 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.100687 12978 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.133121 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.133893 10307 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:31.136723 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 72013.7us
I20240507 08:06:31.078338 12892 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.146556 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.149667 13053 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.151527 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.152572 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.127190 12898 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:31.153517 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.124624 12944 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.156236 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.132376 13049 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:31.158162 10191 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:31.163331 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.166802 12821 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.167368 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:31.168718 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:31.137827 13003 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.174381 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.159930 12867 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:31.118800 13074 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.129108 12980 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:31.179016 13082 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20240507 08:06:31.180140 12980 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.186249 10333 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:31.186818 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.187119 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.190503 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.191411 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.187466 10338 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:31.110203 13065 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.190742 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.166807 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.193743 10156 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:31.196854 10186 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:31.206400 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.209776 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.211347 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:31.214062 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.215485 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.217695 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.223870 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.216821 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.225903 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.207579 12980 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:31.206681 13077 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:31.241389 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.244206 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.236383 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.239904 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:31.246289 10338 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 4: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:31.240649 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.250988 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.251201 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.212863 13002 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.166817 13041 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.204773 13061 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.213313 12959 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.207346 13050 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.255678 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.357286 13003 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:31.383961 13003 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.384588 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:31.306388 12926 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:31.390846 13088 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20240507 08:06:31.287400 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.393447 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.394598 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.395535 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.396176 10333 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:31.396998 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.310907 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.282939 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:31.399420 10191 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:31.400877 10337 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:31.402057 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 126569us
I20240507 08:06:31.385353 10332 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:31.395177 10336 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:31.438067 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:31.438757 10335 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:31.360641 12892 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.401962 13019 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:31.440783 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.441830 13061 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:31.442705 10341 raft_consensus.cc:2379] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:31.400866 10334 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240507 08:06:31.228857 13074 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.243068 12980 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:31.239509 12952 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:31.283725 13011 consensus_meta.cc:287] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.620s user 0.004s sys 0.006s
I20240507 08:06:31.446875 10306 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:31.241820 13077 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:31.278795 12979 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.450629 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.451021 10340 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:31.613893 12980 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:31.614492 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.451159 10940 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:31.567396 10335 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
W20240507 08:06:31.616591 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 169916us
I20240507 08:06:31.617329 10306 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:31.530051 10332 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:31.524680 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1888) took 2092 ms. Trace:
I20240507 08:06:31.618784 10266 rpcz_store.cc:276] 0507 08:06:29.432388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.276702 (+844314us) service_pool.cc:225] Handling call
0507 08:06:30.276954 (+ 252us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:30.776797 (+499843us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:31.524423 (+747626us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b5400116a54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:31.524597 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1002726784}
I20240507 08:06:31.497256 10191 raft_consensus.cc:2454] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:31.504807 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.486831 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.620297 13077 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:31.515873 10264 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39118 (request call id 425) took 1676 ms (client timeout 2074 ms). Trace:
I20240507 08:06:31.512053 12898 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:31.621726 10264 rpcz_store.cc:269] 0507 08:06:29.839077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.057363 (+218286us) service_pool.cc:225] Handling call
0507 08:06:31.515686 (+1458323us) spinlock_profiling.cc:243] Waited 76.3 ms on lock 0x7b54001d2c54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:31.515791 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175399680}
I20240507 08:06:31.622561 10184 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:31.523029 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.623075 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.623908 10338 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:31.621506 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.382655 13065 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:31.504398 13047 consensus_meta.cc:287] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.665s user 0.006s sys 0.011s
I20240507 08:06:31.507977 13041 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.624293 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.361071 13062 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.626453 10906 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:31.625633 10306 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:31.660725 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.629081 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:31.632623 10908 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:31.627200 13061 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240507 08:06:31.634007 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:31.633095 12867 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:31.653105 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:31.626791 13050 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.647178 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.668828 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.669829 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.670118 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.659461 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.659902 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.671350 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.671634 10264 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:31.661821 10306 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:31.662426 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.673228 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.672730 10265 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:31.674039 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:31.674304 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.663079 10186 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 4: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:31.632539 13003 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:31.612650 12959 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:31.658870 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.671977 10262 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:31.443320 12926 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.661319 10258 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:31.663888 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.675979 10306 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:31.676262 13065 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.652940 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:31.679698 10233 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:31.681607 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:31.682452 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.683073 10267 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:31.683868 10937 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240507 08:06:31.685382 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.685806 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:31.686007 10259 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:31.686360 10261 raft_consensus.cc:3032] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:31.686640 10261 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:31.667519 12912 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:31.672719 10941 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:31.626677 13011 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.676788 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:31.665556 13059 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:31.523357 12976 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.666647 12867 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:31.688910 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.690937 10906 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:31.506198 13042 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.681826 12986 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:31.692396 13050 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:31.733255 10306 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:31.779469 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 46320.3us
I20240507 08:06:31.735074 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.780388 10939 raft_consensus.cc:2379] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 9: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:31.659897 13047 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.800236 12731 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.801079 10339 raft_consensus.cc:1226] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:31.800928 12731 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.802160 13047 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.803205 12731 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.803709 12731 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:31.734103 12959 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240507 08:06:31.688268 13049 consensus_meta.cc:287] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.516s user 0.013s sys 0.006s
W20240507 08:06:31.759866 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 69080us
I20240507 08:06:31.706565 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.734459 10261 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.819603 10261 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:31.695760 12979 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:31.766386 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.669854 13041 raft_consensus.cc:2781] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:31.777065 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.857988 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.777877 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:31.859732 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.692931 12912 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:31.798569 13061 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:31.897435 10942 raft_consensus.cc:1226] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:31.838760 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.839697 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.899324 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.900115 10336 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:31.900167 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.841023 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.905002 13041 raft_consensus.cc:686] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [term 8 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:31.741168 12898 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.905251 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:31.741677 12913 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.906352 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:31.906047 13041 consensus_queue.cc:237] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.906976 10186 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:31.907377 12912 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:31.908762 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.909164 12976 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:31.909502 10267 raft_consensus.cc:2379] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:31.804131 12731 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:31.762912 12867 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:31.758559 13059 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:31.764950 12968 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:31.941551 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.942414 10337 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:31.914737 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.919950 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.946133 10940 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 332) took 1442 ms. Trace:
W20240507 08:06:31.921018 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 54408.9us
I20240507 08:06:31.898831 10266 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:31.687150 13065 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.946477 10940 rpcz_store.cc:276] 0507 08:06:30.503258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.503632 (+ 374us) service_pool.cc:225] Handling call
0507 08:06:30.503890 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:31.616331 (+1112441us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:31.945917 (+329586us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b5400412654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:31.946078 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2173613312}
I20240507 08:06:31.952183 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.807103 13050 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:31.958132 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:31.959830 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.855361 12959 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:31.856551 12979 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:31.964282 10264 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:31.969640 10261 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:31.973747 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:31.973806 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:31.974457 10337 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:31.951589 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:31.948057 12968 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:31.941797 13059 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:31.906070 12913 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.966799 13050 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:31.994706 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:31.898638 13049 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:31.971132 12959 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:31.973251 12979 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:32.008697 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:32.016661 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.017889 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:32.019136 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.020356 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.020310 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:32.021224 10191 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240507 08:06:32.022240 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.022929 10191 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
W20240507 08:06:32.026274 13105 log.cc:924] Time spent T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20240507 08:06:32.034411 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.035434 10336 raft_consensus.cc:2379] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 8: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:32.038470 13042 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.043134 13049 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.044194 10183 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:32.046759 12731 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:32.054623 10186 raft_consensus.cc:2454] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:32.055992 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:32.056982 10939 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43936 (request call id 208) took 2193 ms (client timeout 1871 ms). Trace:
I20240507 08:06:31.994036 12968 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:31.973018 13061 consensus_queue.cc:1046] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20240507 08:06:32.060741 10186 raft_consensus.cc:2396] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20240507 08:06:32.057946 10939 rpcz_store.cc:269] 0507 08:06:29.864093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:29.882090 (+ 17997us) service_pool.cc:225] Handling call
0507 08:06:30.197163 (+315073us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b58002f2e98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:06:32.056697 (+1859534us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54004114d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.056898 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3005900032}
I20240507 08:06:32.061668 10908 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:32.045372 13011 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:31.999177 13003 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:32.067194 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 386168us
I20240507 08:06:32.068179 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.071306 10264 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.071422 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.069494 10191 raft_consensus.cc:2454] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 9.
I20240507 08:06:32.072875 10186 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:32.074096 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.073467 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.077899 10908 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.075810 10191 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:32.081787 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.083204 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.084241 10936 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
W20240507 08:06:32.084642 10233 consensus_peers.cc:485] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 -> Peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849): Couldn't send request to peer 5d6656a616b747d393ff3de5328d2540. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c898839b7c84ce6ba0e135f778ad72d. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:06:32.073834 13003 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.085203 13047 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:32.085754 10233 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:32.085712 13047 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.086737 13003 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.087585 13047 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:32.059484 12867 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.089795 13065 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.090991 12913 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.091153 12867 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.092025 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.092147 10935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 334) took 1400 ms. Trace:
I20240507 08:06:32.092550 10935 rpcz_store.cc:276] 0507 08:06:30.691729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.699270 (+ 7541us) service_pool.cc:225] Handling call
0507 08:06:30.721365 (+ 22095us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:31.102376 (+381011us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.091935 (+989559us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540067ff54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:32.092091 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457420032}
I20240507 08:06:32.096261 13087 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:32.096087 12959 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.099188 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.096757 13087 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.100639 10336 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:32.101063 13087 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:32.103786 13098 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.850s user 0.003s sys 0.008s
I20240507 08:06:32.105612 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.107332 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.109211 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.110183 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.111591 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.108160 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.107708 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1706) took 1022 ms. Trace:
I20240507 08:06:32.112627 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.113263 10186 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 9.
I20240507 08:06:32.112989 10184 rpcz_store.cc:276] 0507 08:06:31.084995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:31.114914 (+ 29919us) service_pool.cc:225] Handling call
0507 08:06:32.107523 (+992609us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54000479d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.107650 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1165114240}
W20240507 08:06:32.112542 10192 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1681) took 1603 ms (client timeout 1505 ms). Trace:
W20240507 08:06:32.113903 10192 rpcz_store.cc:269] 0507 08:06:30.509229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.509453 (+ 224us) service_pool.cc:225] Handling call
0507 08:06:32.112315 (+1602862us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b54001bf3d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.112441 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1268882688}
I20240507 08:06:32.114240 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.114341 13042 raft_consensus.cc:2781] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:32.114863 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.115475 10192 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.118243 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1684) took 1579 ms. Trace:
I20240507 08:06:32.083950 10191 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:32.118607 10185 rpcz_store.cc:276] 0507 08:06:30.539062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.539588 (+ 526us) service_pool.cc:225] Handling call
0507 08:06:32.118090 (+1578502us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54001405d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.118189 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":953775872}
I20240507 08:06:32.120740 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 345) took 1277 ms. Trace:
I20240507 08:06:32.121115 10339 rpcz_store.cc:276] 0507 08:06:30.843113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.843463 (+ 350us) service_pool.cc:225] Handling call
0507 08:06:30.903575 (+ 60112us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:31.801462 (+897887us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.120549 (+319087us) spinlock_profiling.cc:243] Waited 898 ms on lock 0x7b540013f454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:32.120687 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2064619264}
I20240507 08:06:32.114935 13042 raft_consensus.cc:686] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [term 9 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:32.115500 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.125792 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.133116 10185 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20240507 08:06:32.126020 10232 leader_election.cc:334] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.505s (SENT)
I20240507 08:06:32.133730 10232 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:32.127074 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 53178.9us
I20240507 08:06:32.074420 13011 raft_consensus.cc:2781] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:32.144584 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.148375 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.149900 10264 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:32.075387 12926 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.146181 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.163255 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.140050 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.140978 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.145329 12998 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20240507 08:06:32.163787 10185 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:32.179036 10185 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:32.180342 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.181922 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:32.184360 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.182610 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.184892 10185 raft_consensus.cc:2396] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20240507 08:06:32.185597 10184 raft_consensus.cc:2396] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240507 08:06:32.186769 12926 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:32.187283 12926 raft_consensus.cc:686] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.187423 10936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43924 (request call id 333) took 1678 ms. Trace:
I20240507 08:06:32.188139 12926 consensus_queue.cc:237] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.189144 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.189214 12979 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:32.189975 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1702) took 1446 ms (client timeout 1582 ms). Trace:
W20240507 08:06:32.190371 10190 rpcz_store.cc:269] 0507 08:06:30.743987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.762900 (+ 18913us) service_pool.cc:225] Handling call
0507 08:06:32.189013 (+1426113us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800002a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:06:32.189917 (+ 904us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26126592}
I20240507 08:06:32.190909 13046 consensus_queue.cc:1046] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:32.191274 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.191576 12979 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:32.192174 12912 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.192991 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.193568 10190 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:32.194308 12912 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.194356 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.195200 10190 raft_consensus.cc:2379] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:32.195839 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.196725 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.199195 10190 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.199249 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.200543 12986 raft_consensus.cc:3032] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:32.200896 12986 raft_consensus.cc:729] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.192179 12979 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:32.187785 10936 rpcz_store.cc:276] 0507 08:06:30.508982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.623808 (+114826us) service_pool.cc:225] Handling call
0507 08:06:32.070074 (+1446266us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.084696 (+ 14622us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.187212 (+102516us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54004114d0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.187363 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3237617408}
I20240507 08:06:32.198549 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.201653 12986 consensus_queue.cc:260] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.202875 12986 raft_consensus.cc:3037] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.202833 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.204011 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.205001 10184 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240507 08:06:32.205283 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1617) took 1036 ms. Trace:
I20240507 08:06:32.210826 10341 rpcz_store.cc:276] 0507 08:06:31.168941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:31.205352 (+ 36411us) service_pool.cc:225] Handling call
0507 08:06:32.205102 (+999750us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54003c4954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.205232 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":515508096}
I20240507 08:06:32.205997 12959 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.206146 10338 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.205986 10184 raft_consensus.cc:1226] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:32.206620 12979 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:32.207787 12958 raft_consensus.cc:3037] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:32.219126 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.219476 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1694) took 1547 ms. Trace:
I20240507 08:06:32.219867 10184 rpcz_store.cc:276] 0507 08:06:30.671901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.205469 (+1533568us) service_pool.cc:225] Handling call
0507 08:06:32.205731 (+ 262us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.217869 (+ 12138us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.219427 (+ 1558us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.220806 10943 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43910 (request call id 233) took 2431 ms. Trace:
I20240507 08:06:32.221271 10943 rpcz_store.cc:276] 0507 08:06:29.789590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:29.790215 (+ 625us) service_pool.cc:225] Handling call
0507 08:06:29.790465 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:31.136804 (+1346339us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.220541 (+1083737us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400810454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:32.220737 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3069604096}
I20240507 08:06:32.221824 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.224028 10183 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:32.224802 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1700) took 1551 ms. Trace:
I20240507 08:06:32.221962 12979 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:32.226557 12786 raft_consensus.cc:3032] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:32.226946 12786 raft_consensus.cc:729] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
W20240507 08:06:32.227883 13077 consensus_meta.cc:287] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.606s user 0.009s sys 0.000s
I20240507 08:06:32.227751 12786 consensus_queue.cc:260] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.228242 13077 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.232621 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.233417 10191 raft_consensus.cc:2396] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240507 08:06:32.225863 10184 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.234107 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 319) took 1551 ms. Trace:
W20240507 08:06:32.234571 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 52913.9us
I20240507 08:06:32.235455 10155 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.235594 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.234438 10184 rpcz_store.cc:276] 0507 08:06:30.683125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.225413 (+1542288us) service_pool.cc:225] Handling call
0507 08:06:32.225696 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.233907 (+ 8211us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.234061 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.236191 13060 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:32.236563 10155 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.237007 13060 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:32.225235 10183 rpcz_store.cc:276] 0507 08:06:30.673495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.223459 (+1549964us) service_pool.cc:225] Handling call
0507 08:06:32.223810 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.224529 (+ 719us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.224739 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.239245 10183 raft_consensus.cc:1226] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:32.239594 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2087) took 1175 ms. Trace:
I20240507 08:06:32.239723 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2089) took 1154 ms. Trace:
I20240507 08:06:32.240172 10183 rpcz_store.cc:276] 0507 08:06:31.085366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.238857 (+1153491us) service_pool.cc:225] Handling call
0507 08:06:32.239103 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.239544 (+ 441us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.239680 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.240826 10189 raft_consensus.cc:2454] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:32.242018 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1708) took 1084 ms. Trace:
I20240507 08:06:32.242638 10183 rpcz_store.cc:276] 0507 08:06:31.157480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.241156 (+1083676us) service_pool.cc:225] Handling call
0507 08:06:32.241414 (+ 258us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.241565 (+ 151us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:32.241614 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:32.241667 (+ 53us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:32.241714 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.241783 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:32.241951 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.244603 10183 raft_consensus.cc:1226] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.245283 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2094) took 1076 ms. Trace:
I20240507 08:06:32.183794 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.184080 10192 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:32.245671 10183 rpcz_store.cc:276] 0507 08:06:31.168738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.244047 (+1075309us) service_pool.cc:225] Handling call
0507 08:06:32.244403 (+ 356us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.245031 (+ 628us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.245229 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.151945 13011 raft_consensus.cc:686] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [term 6 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:32.254539 13011 consensus_queue.cc:237] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.233172 12979 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.246402 10184 rpcz_store.cc:276] 0507 08:06:31.064953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.238722 (+1173769us) service_pool.cc:225] Handling call
0507 08:06:32.239024 (+ 302us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.239214 (+ 190us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:32.239251 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:32.239304 (+ 53us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:32.239357 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.239428 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:32.239538 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.234640 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2075) took 1475 ms. Trace:
I20240507 08:06:32.249879 10192 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:32.142753 12913 consensus_queue.cc:1046] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20240507 08:06:32.260964 10184 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.259035 10191 rpcz_store.cc:276] 0507 08:06:30.759348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.233961 (+1474613us) service_pool.cc:225] Handling call
0507 08:06:32.234195 (+ 234us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.234347 (+ 152us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:32.234370 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:32.234416 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:32.234454 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.234504 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:32.234593 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.279237 10190 raft_consensus.cc:2454] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 9.
I20240507 08:06:32.262579 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2095) took 1090 ms. Trace:
W20240507 08:06:32.285562 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47348.5us
I20240507 08:06:32.286223 10908 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:32.286470 10155 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.285604 10192 rpcz_store.cc:276] 0507 08:06:31.171907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.249288 (+1077381us) service_pool.cc:225] Handling call
0507 08:06:32.249640 (+ 352us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.262322 (+ 12682us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.262525 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.289413 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 321) took 1093 ms. Trace:
I20240507 08:06:32.289921 10184 rpcz_store.cc:276] 0507 08:06:31.195694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.260431 (+1064737us) service_pool.cc:225] Handling call
0507 08:06:32.260760 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.284516 (+ 23756us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.289360 (+ 4844us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.287892 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.181123 13049 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:32.294175 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.291262 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1711) took 1050 ms. Trace:
W20240507 08:06:32.291394 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 248901us
I20240507 08:06:32.237525 13060 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.292833 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.301753 10185 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.294916 10306 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:32.298446 10186 raft_consensus.cc:1226] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:32.259279 13076 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20240507 08:06:32.308401 13076 raft_consensus.cc:3032] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:32.308902 13076 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
W20240507 08:06:32.296113 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 206794us
I20240507 08:06:32.310801 10906 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:32.309435 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 215093us
I20240507 08:06:32.201216 12968 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.162634 13111 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.146s
I20240507 08:06:32.368786 13111 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:32.326455 10186 raft_consensus.cc:1226] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:32.372627 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.373363 10186 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:32.314725 12807 consensus_queue.cc:1046] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
I20240507 08:06:32.374722 12807 raft_consensus.cc:3032] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:32.375030 12807 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.259588 13053 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:32.294335 13014 raft_consensus.cc:2781] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:32.302577 12958 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:32.322240 13043 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:32.298856 10190 rpcz_store.cc:276] 0507 08:06:31.241383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.288361 (+1046978us) service_pool.cc:225] Handling call
0507 08:06:32.288742 (+ 381us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.288957 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:32.288991 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:32.289033 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:32.289076 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.289130 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:32.291216 (+ 2086us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:32.420805 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.367558 13095 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240507 08:06:32.348776 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 61015.9us
I20240507 08:06:32.422040 10155 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.347730 12733 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:32.350756 12979 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:32.383239 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1686) took 1722 ms. Trace:
I20240507 08:06:32.423064 10187 rpcz_store.cc:276] 0507 08:06:30.661330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.661544 (+ 214us) service_pool.cc:225] Handling call
0507 08:06:30.661962 (+ 418us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:30.740737 (+ 78775us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:30.740783 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:31.513795 (+773012us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b2c00000918. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8f809f11 00007fcf8f819542 00007fcf90e62fc1 00007fcf90e277da 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782
0507 08:06:31.514186 (+ 391us) log.cc:842] Serialized 26 byte log entry
0507 08:06:31.659089 (+144903us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:31.962737 (+303648us) log.cc:842] Serialized 14 byte log entry
0507 08:06:31.963020 (+ 283us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.378598 (+415578us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b1800394780. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8decab06 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089
0507 08:06:32.382935 (+ 4337us) spinlock_profiling.cc:243] Waited 78.3 ms on lock 0x7b54003b4054. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:32.382981 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:32.383034 (+ 53us) raft_consensus.cc:1671] finished
0507 08:06:32.383048 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:32.383181 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":356137,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":1128,"raft.queue_time_us":24515,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":4579,"spinlock_wait_cycles":236819840,"thread_start_us":432366,"threads_started":2,"wal-append.queue_time_us":264920}
I20240507 08:06:32.386346 12968 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.391973 10186 raft_consensus.cc:1226] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:32.393707 10304 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:32.394855 13076 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.323907 12959 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
W20240507 08:06:32.397506 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1696) took 1725 ms (client timeout 2066 ms). Trace:
W20240507 08:06:32.431547 10189 rpcz_store.cc:269] 0507 08:06:30.672444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:31.167126 (+494682us) service_pool.cc:225] Handling call
0507 08:06:32.397309 (+1230183us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400082ad4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:32.397438 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2434658176}
I20240507 08:06:32.356176 13064 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:32.394855 10089 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59954 (request call id 218) took 5077 ms. Trace:
I20240507 08:06:32.432085 10155 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.353471 12838 consensus_queue.cc:1046] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
W20240507 08:06:32.398766 12980 consensus_meta.cc:287] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.768s user 0.009s sys 0.000s
I20240507 08:06:32.262264 13102 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20240507 08:06:32.399688 13077 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.315851 13042 consensus_queue.cc:237] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.349421 12867 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
W20240507 08:06:32.406498 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 280645us
W20240507 08:06:32.408100 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 250390us
I20240507 08:06:32.409032 10906 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.354918 13051 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:32.435215 13061 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:32.417397 12958 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:32.363729 10185 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:32.470067 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.426167 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.405819 13086 consensus_queue.cc:1046] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20240507 08:06:32.475626 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.476997 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:32.429083 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 64970.1us
I20240507 08:06:32.478247 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:32.419500 13043 raft_consensus.cc:686] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.433970 10155 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:32.480417 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 46539.9us
I20240507 08:06:32.481328 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.435231 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.436012 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.437523 10264 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:32.417076 13053 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:32.440258 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.483664 10190 raft_consensus.cc:2379] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:32.442313 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.484580 10266 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.433151 12681 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:32.434643 12952 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:32.444638 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.492455 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:32.492588 10186 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:32.449999 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.479533 12907 consensus_queue.cc:1046] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20240507 08:06:32.495589 12907 raft_consensus.cc:3032] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:32.495918 12907 raft_consensus.cc:729] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:32.496592 12907 consensus_queue.cc:260] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.497666 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:32.474953 12958 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.498621 12959 raft_consensus.cc:3032] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:32.499013 12959 raft_consensus.cc:729] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.466313 13076 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.427165 13014 raft_consensus.cc:686] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [term 9 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:32.501422 12731 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:32.486414 13053 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.502282 13014 consensus_queue.cc:237] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.458086 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.415419 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.505335 10337 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:32.506508 13050 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.486599 12681 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:32.508613 13050 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.479359 12979 consensus_queue.cc:1046] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20240507 08:06:32.508935 13043 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.427481 13095 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:32.460228 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:32.438446 13109 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.432466 10089 rpcz_store.cc:276] 0507 08:06:27.317641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:27.317816 (+ 175us) service_pool.cc:225] Handling call
0507 08:06:32.394788 (+5076972us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:30.502453 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:30.514616 (+ 12163us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:30.514658 (+ 42us) write_op.cc:435] Acquired schema lock
0507 08:06:30.514677 (+ 19us) tablet.cc:660] Decoding operations
0507 08:06:30.515557 (+ 880us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:30.515633 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:30.515664 (+ 31us) tablet.cc:683] Acquiring locks for 7 operations
0507 08:06:30.516285 (+ 621us) tablet.cc:699] Row locks acquired
0507 08:06:30.516305 (+ 20us) write_op.cc:260] PREPARE: finished
0507 08:06:30.516458 (+ 153us) write_op.cc:270] Start()
0507 08:06:30.516573 (+ 115us) write_op.cc:276] Timestamp: P: 1715069190516406 usec, L: 0
0507 08:06:30.516595 (+ 22us) op_driver.cc:331] REPLICATION: starting
0507 08:06:30.641429 (+124834us) log.cc:842] Serialized 2870 byte log entry
0507 08:06:31.330721 (+689292us) op_driver.cc:444] REPLICATION: finished
0507 08:06:31.493967 (+163246us) write_op.cc:301] APPLY: starting
0507 08:06:31.494049 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:31.494257 (+ 208us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:31.494390 (+ 133us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:31.504283 (+ 9893us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:31.504513 (+ 230us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0507 08:06:31.504562 (+ 49us) write_op.cc:312] APPLY: finished
0507 08:06:31.504824 (+ 262us) log.cc:842] Serialized 59 byte log entry
0507 08:06:31.504994 (+ 170us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:31.505119 (+ 125us) write_op.cc:454] Released schema lock
0507 08:06:31.505167 (+ 48us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:31.505422 (+ 255us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:31.505451 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2223951,"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":135987,"apply.run_cpu_time_us":3141,"apply.run_wall_time_us":70720,"num_ops":7,"prepare.queue_time_us":501788,"prepare.run_cpu_time_us":7636,"prepare.run_wall_time_us":206513,"raft.queue_time_us":172417,"raft.run_cpu_time_us":6444,"raft.run_wall_time_us":135114,"replication_time_us":809855,"thread_start_us":439328,"threads_started":4,"wal-append.queue_time_us":106825}]]}
W20240507 08:06:32.456878 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47944.3us
I20240507 08:06:32.484530 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.519874 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.487704 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.521152 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.521322 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.464555 13061 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.499830 12959 consensus_queue.cc:260] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.493050 10261 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
W20240507 08:06:32.455853 10232 leader_election.cc:334] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.641s (SENT)
I20240507 08:06:32.464083 13051 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:32.463936 13003 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.434926 13064 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.517102 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.489286 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.577075 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:32.486698 12968 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:32.442034 12980 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:32.569876 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 114119us
W20240507 08:06:32.614004 10232 leader_election.cc:334] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.665s (SENT)
I20240507 08:06:32.614543 10232 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.505518 13047 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:32.521251 10266 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
W20240507 08:06:32.577984 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 88846.9us
I20240507 08:06:32.623728 13121 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.623911 10155 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.624640 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:32.625833 10155 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.568344 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.517900 13095 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:32.510183 13059 consensus_meta.cc:287] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.502s user 0.008s sys 0.001s
I20240507 08:06:32.568624 12907 raft_consensus.cc:3037] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:32.581070 13050 consensus_queue.cc:1046] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20240507 08:06:32.659919 10185 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:32.660836 12980 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.577807 12912 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20240507 08:06:32.661677 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:32.662303 10938 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:32.663247 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:32.612264 12913 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:32.665921 12913 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:32.666342 12913 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.667423 13109 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:32.459945 12807 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.668967 12807 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.671510 13061 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.672670 13061 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.493427 13049 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:32.533282 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.492794 12838 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.508636 12681 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.576854 13051 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:32.618219 10190 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20240507 08:06:32.621608 10232 leader_election.cc:334] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795): Timed out: RequestConsensusVote RPC to 127.8.213.129:32795 timed out after 1.582s (SENT)
W20240507 08:06:32.694128 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 72613.6us
I20240507 08:06:32.695097 13063 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:32.511632 12731 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:32.698343 12731 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.516747 12979 raft_consensus.cc:3032] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:32.635746 10155 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:32.580488 10158 connection.cc:232] RPC call timeout handler was delayed by -1.2843532059999916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20240507 08:06:32.623250 13047 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:32.649684 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:32.650848 10906 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.616576 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:32.686283 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.700592 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:32.687683 10337 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:32.718385 10333 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.718845 10158 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:32.705267 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 54525.3us
I20240507 08:06:32.719647 10232 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:32.718295 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 82704.7us
I20240507 08:06:32.674533 10938 raft_consensus.cc:1226] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.719944 10158 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.705559 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:32.720459 10155 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.722312 10906 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.731156 10906 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.732719 10906 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.722934 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:32.739868 10158 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:32.743669 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 385925us
I20240507 08:06:32.699066 12913 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.747169 10185 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.749184 12913 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.703713 12959 raft_consensus.cc:3037] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:32.622834 12968 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.755935 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:32.702551 12979 raft_consensus.cc:729] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:32.717864 13047 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:32.725208 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2098) took 1292 ms. Trace:
I20240507 08:06:32.721170 13045 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:32.723719 12952 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election won for term 4
I20240507 08:06:32.686185 13059 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:32.737654 13119 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:32.804119 10943 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:32.756933 10158 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.758370 12998 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:32.759737 13083 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:32.758980 13003 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.757962 13109 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.707047 13053 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.760686 13120 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:32.768429 13049 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:32.821703 13059 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.823132 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:32.848173 13049 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:32.848732 10337 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:32.829977 13014 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:32.814185 13064 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:32.832509 12979 consensus_queue.cc:260] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.849347 13049 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.850457 12979 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:32.849959 13014 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.833807 10338 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.802647 13089 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:32.816543 12838 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:32.847311 10942 raft_consensus.cc:1226] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:32.814381 13117 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240507 08:06:32.875213 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 371956us
I20240507 08:06:32.818014 12998 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:32.880796 12998 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:32.882517 13108 raft_consensus.cc:3032] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:32.882930 13108 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:32.883742 13108 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.885056 13108 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:32.815686 13119 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:32.880813 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:32.886579 10185 raft_consensus.cc:595] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.888276 13062 raft_consensus.cc:3032] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:32.888684 13062 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:32.886049 13119 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:32.891850 13132 ts_tablet_manager.cc:902] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:32.920619 13063 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:32.921110 13063 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:32.889441 13062 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:32.924726 13062 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:32.929551 13116 raft_consensus.cc:3032] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:32.930063 13116 raft_consensus.cc:729] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:32.931015 13116 consensus_queue.cc:260] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.937923 13019 raft_consensus.cc:3032] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:32.938429 13019 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
W20240507 08:06:32.956980 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 496578us
I20240507 08:06:32.857997 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:32.957952 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 77731.6us
I20240507 08:06:32.958287 10185 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240507 08:06:32.958685 10185 pending_rounds.cc:76] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 5
I20240507 08:06:32.958979 10185 pending_rounds.cc:97] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20240507 08:06:32.958751 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 100903us
I20240507 08:06:32.959656 10185 raft_consensus.cc:2866] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20240507 08:06:32.960613 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 240334us
I20240507 08:06:32.960952 10185 raft_consensus.cc:1261] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:32.961483 10155 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.963050 10304 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:32.963726 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 206981us
I20240507 08:06:32.964596 10158 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.966255 10333 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:32.968600 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.974448 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.974794 10158 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:32.975337 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:32.986712 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:32.987833 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:32.971673 10087 catalog_manager.cc:5484] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:32.991251 10087 catalog_manager.cc:5484] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 7, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:32.993266 13132 tablet_copy_client.cc:316] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: tablet copy: Beginning tablet copy session from remote peer at address 127.8.213.130:41081
I20240507 08:06:32.993429 10087 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 2 to 5, leader changed from 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:32.997848 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.000576 10156 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:33.002961 10306 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:33.004839 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 285373us
I20240507 08:06:33.005823 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.006281 10232 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:33.001880 10087 catalog_manager.cc:5484] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:33.005836 10337 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:33.007269 10232 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:33.009727 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:32.939390 13019 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:32.970613 12731 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:33.011500 10158 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.011736 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2099) took 1578 ms. Trace:
I20240507 08:06:33.010936 13142 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:32.969409 13066 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:32.984336 13102 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:32.987586 13116 raft_consensus.cc:3037] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:33.004628 13053 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.013741 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.013648 10232 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:33.036717 10232 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:33.036936 10304 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.011240 13121 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:33.011090 12986 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.022594 13092 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:33.022660 13143 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:32.984557 13083 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:33.023980 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2097) took 1592 ms. Trace:
I20240507 08:06:32.996917 13059 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:32.997604 12838 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:33.006906 12952 raft_consensus.cc:686] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.023989 10191 rpcz_store.cc:276] 0507 08:06:31.433106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.291747 (+858641us) service_pool.cc:225] Handling call
0507 08:06:32.292063 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.292306 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 8
0507 08:06:32.738811 (+446505us) log.cc:842] Serialized 14 byte log entry
0507 08:06:33.010827 (+272016us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:33.011363 (+ 536us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:33.011442 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:33.011532 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:33.011669 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":118420,"prepare.queue_time_us":102580,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"thread_start_us":599279,"threads_started":2,"wal-append.queue_time_us":255318}
I20240507 08:06:33.011122 13115 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20240507 08:06:33.031591 10338 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:32.987419 12786 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.009472 12926 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:33.007422 10185 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20240507 08:06:33.043284 13019 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.014236 13050 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240507 08:06:33.029292 13076 consensus_meta.cc:287] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.002s
W20240507 08:06:33.044224 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 299480us
I20240507 08:06:33.045720 10232 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.081983 12968 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:33.027693 13089 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.065977 10192 rpcz_store.cc:276] 0507 08:06:31.431781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.290427 (+858646us) service_pool.cc:225] Handling call
0507 08:06:32.291618 (+ 1191us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.292061 (+ 443us) raft_consensus.cc:1520] Early marking committed up to index 3
0507 08:06:32.703637 (+411576us) log.cc:842] Serialized 14 byte log entry
0507 08:06:33.023536 (+319899us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:33.023610 (+ 74us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:33.023678 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:33.023767 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:33.023914 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95390,"prepare.queue_time_us":102648,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":30241,"spinlock_wait_cycles":10864384,"thread_start_us":600785,"threads_started":2,"wal-append.queue_time_us":292611}
I20240507 08:06:33.025926 13142 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:33.027837 13065 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:33.028268 13102 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:33.072182 13139 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.072346 10277 tablet_copy_service.cc:140] P bcf08c8276ba4a9ba646ad0f7090fcd8: Received BeginTabletCopySession request for tablet 0c898839b7c84ce6ba0e135f778ad72d from peer 5d6656a616b747d393ff3de5328d2540 ({username='slave'} at 127.0.0.1:39118)
I20240507 08:06:33.063030 13125 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:32.771207 10184 rpcz_store.cc:276] 0507 08:06:31.432473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.290724 (+858251us) service_pool.cc:225] Handling call
0507 08:06:32.290987 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.291179 (+ 192us) raft_consensus.cc:1520] Early marking committed up to index 6
0507 08:06:32.348931 (+ 57752us) log.cc:842] Serialized 14 byte log entry
0507 08:06:32.724903 (+375972us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:32.724951 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:32.725004 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:32.725061 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:32.725158 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4451,"prepare.queue_time_us":103312,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":17281,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":14786,"thread_start_us":370870,"threads_started":1}
I20240507 08:06:33.228276 13102 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
W20240507 08:06:33.082216 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 71696.6us
I20240507 08:06:33.013432 13086 consensus_queue.cc:1046] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20240507 08:06:33.023555 12980 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:33.044660 13066 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:33.100129 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.044847 13053 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:33.046105 12981 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:33.114068 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:33.050493 10304 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:33.184118 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 138534us
W20240507 08:06:33.298383 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 248044us
I20240507 08:06:33.299068 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.299356 10232 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.300148 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.187215 10231 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:33.301280 13043 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240507 08:06:33.184803 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 209626us
I20240507 08:06:33.302685 13063 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.304642 13063 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.106261 13092 raft_consensus.cc:686] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:33.305956 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.306705 13092 consensus_queue.cc:237] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.307263 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.309298 10263 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:33.184731 10185 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.309942 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.310886 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.311313 10261 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:33.312125 13047 consensus_meta.cc:287] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.510s user 0.002s sys 0.022s
I20240507 08:06:33.312384 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.312515 13047 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.312880 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.313503 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.313923 10185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:33.314375 13047 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:33.314437 10185 raft_consensus.cc:595] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.314749 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.090148 13143 raft_consensus.cc:686] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.315909 10185 raft_consensus.cc:1261] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240507 08:06:33.316349 13143 consensus_queue.cc:237] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.317188 10185 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20240507 08:06:33.317303 10333 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240507 08:06:33.149639 13142 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:33.143442 13112 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:33.116246 13014 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.105355 12926 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.007810 10087 catalog_manager.cc:5484] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
W20240507 08:06:33.320844 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 135638us
I20240507 08:06:33.239738 10940 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:33.240450 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2114) took 1138 ms. Trace:
I20240507 08:06:33.323503 10189 rpcz_store.cc:276] 0507 08:06:32.102467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.703141 (+600674us) service_pool.cc:225] Handling call
0507 08:06:32.724060 (+ 20919us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:32.724277 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 4
0507 08:06:32.767924 (+ 43647us) log.cc:842] Serialized 14 byte log entry
0507 08:06:33.240162 (+472238us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:33.240208 (+ 46us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:33.240250 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:33.240309 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:33.240403 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4482,"prepare.queue_time_us":31908,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":8503,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":16635,"thread_start_us":471826,"threads_started":2,"wal-append.queue_time_us":331842}
I20240507 08:06:33.246496 10259 raft_consensus.cc:2379] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 8: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:33.227913 13065 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:33.250408 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.250355 13115 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240507 08:06:33.243804 10277 tablet_copy_service.cc:161] P bcf08c8276ba4a9ba646ad0f7090fcd8: Beginning new tablet copy session on tablet 0c898839b7c84ce6ba0e135f778ad72d from peer 5d6656a616b747d393ff3de5328d2540 at {username='slave'} at 127.0.0.1:39118: session id = 5d6656a616b747d393ff3de5328d2540-0c898839b7c84ce6ba0e135f778ad72d
I20240507 08:06:33.257985 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.258868 13086 raft_consensus.cc:3032] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:33.259784 10306 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.260339 13059 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:33.257269 10943 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
W20240507 08:06:33.261323 13144 log.cc:924] Time spent T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:06:33.263969 13066 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240507 08:06:33.237133 13125 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:33.265650 12807 consensus_meta.cc:287] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.596s user 0.014s sys 0.010s
I20240507 08:06:33.266460 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:33.265394 13060 consensus_meta.cc:287] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.941s user 0.012s sys 0.000s
W20240507 08:06:33.269398 12681 consensus_meta.cc:287] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.579s user 0.013s sys 0.019s
W20240507 08:06:33.270466 13138 log.cc:924] Time spent T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.241s user 0.000s sys 0.001s
W20240507 08:06:33.271927 13051 consensus_meta.cc:287] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.578s user 0.006s sys 0.013s
W20240507 08:06:33.272749 12958 consensus_meta.cc:287] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.774s user 0.015s sys 0.006s
I20240507 08:06:33.273931 10942 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:33.257388 12998 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:33.275671 12733 consensus_meta.cc:287] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.846s user 0.008s sys 0.029s
W20240507 08:06:33.276871 12907 consensus_meta.cc:287] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.619s user 0.015s sys 0.010s
W20240507 08:06:33.281010 12959 consensus_meta.cc:287] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.511s user 0.016s sys 0.013s
I20240507 08:06:33.283766 12981 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:33.288002 13119 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.266150 13053 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.244073 12952 consensus_queue.cc:237] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.317880 13141 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.309968 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.332581 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.353454 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.353945 10158 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:33.337011 10231 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.334051 13065 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.350328 10908 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.322319 10087 catalog_manager.cc:5484] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 4 to 6, leader changed from 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:33.357883 13042 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20240507 08:06:33.362617 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.363519 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.349836 13086 raft_consensus.cc:729] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:33.358345 12979 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:33.372795 10906 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.374855 10939 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:33.375065 10233 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:33.376449 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 74933.1us
I20240507 08:06:33.377753 10188 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:33.378832 10262 raft_consensus.cc:1261] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20240507 08:06:33.396766 10192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:33.397408 10192 raft_consensus.cc:595] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.397836 10192 raft_consensus.cc:603] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:33.365412 13125 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.382323 10337 raft_consensus.cc:1261] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20240507 08:06:33.383298 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 123629us
I20240507 08:06:33.400094 10260 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:33.385286 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48425.9us
W20240507 08:06:33.314785 13061 consensus_meta.cc:287] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.642s user 0.013s sys 0.010s
W20240507 08:06:33.372856 13095 consensus_meta.cc:287] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.686s user 0.008s sys 0.021s
I20240507 08:06:33.392308 10266 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:33.401599 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.403757 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.404897 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:33.368245 13060 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.369397 12681 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.389266 12907 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:33.381281 13147 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:33.398171 13014 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.383129 12986 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:33.376680 12958 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.407279 13049 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:33.413875 10304 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.411782 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.386535 13133 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:33.415938 10260 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
W20240507 08:06:33.408957 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 49170.8us
I20240507 08:06:33.373900 13051 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.431631 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.398650 12979 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.404091 13111 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:33.372023 13121 consensus_queue.cc:1046] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20240507 08:06:33.417089 13019 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:33.407860 13141 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.427088 10192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0778bf2d68148c495412379f7818b28"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:33.442242 10192 raft_consensus.cc:595] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.442695 10192 raft_consensus.cc:603] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:33.427256 10265 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:33.428124 10183 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.384523 12981 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.415937 12786 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:33.447389 10190 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:33.419353 13147 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.411638 12838 consensus_queue.cc:1046] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20240507 08:06:33.440256 13049 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.409309 13120 consensus_queue.cc:1035] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20240507 08:06:33.447955 10185 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20240507 08:06:33.411103 13115 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20240507 08:06:33.448628 10187 raft_consensus.cc:2379] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:33.448704 10232 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:33.425633 12907 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:33.451225 13060 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.451833 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240507 08:06:33.444337 10334 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
W20240507 08:06:33.453039 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48222.7us
I20240507 08:06:33.455796 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.458287 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.410357 13095 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.438580 12979 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:33.468937 10231 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.471590 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.472425 10262 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:33.469457 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:33.474323 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:33.444979 13111 raft_consensus.cc:686] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:33.489817 13141 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:33.490608 10934 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 7: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:33.490250 13111 consensus_queue.cc:237] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.491199 13051 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.492452 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.493854 13063 consensus_queue.cc:1046] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:33.495003 13063 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:33.495364 13063 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.496162 13063 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.497220 13063 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:33.498044 13139 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.498723 13139 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:33.498986 12958 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:33.501957 13154 log.cc:924] Time spent T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240507 08:06:33.502130 13049 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.503289 13119 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:33.504290 13153 log.cc:924] Time spent T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20240507 08:06:33.504515 12998 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.442294 13019 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.507365 13019 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.509132 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:33.461745 13042 consensus_queue.cc:1035] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20240507 08:06:33.478206 10232 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.510211 12907 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:33.510603 12907 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:33.477133 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 63379.8us
I20240507 08:06:33.440927 13050 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20240507 08:06:33.488864 13095 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:33.509789 10338 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:33.457262 13147 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:33.530910 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 55013.2us
I20240507 08:06:33.512418 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.447490 13156 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:33.514317 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.539844 10261 raft_consensus.cc:2379] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 7: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:33.514694 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.490264 12681 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:33.510917 10232 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.541951 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.527069 10907 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:33.513849 10265 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:33.543784 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:33.510995 12907 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:33.545529 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.512128 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.533470 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.538652 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.529431 10262 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:33.490258 13141 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:33.569613 13159 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:33.511250 12926 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:33.519572 13116 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:33.443650 13053 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:33.575708 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 63676us
I20240507 08:06:33.580837 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:33.582873 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 109721us
I20240507 08:06:33.545040 13156 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.526477 13092 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20240507 08:06:33.572613 13118 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:33.602044 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.575006 13059 raft_consensus.cc:3032] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:33.576301 13141 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.584906 13061 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.617856 12681 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.616835 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1748) took 1216 ms. Trace:
I20240507 08:06:33.588472 13121 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.624470 10187 rpcz_store.cc:276] 0507 08:06:32.400383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.476722 (+ 76339us) service_pool.cc:225] Handling call
0507 08:06:33.616645 (+1139923us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b54003b3654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:33.616774 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2216491264}
I20240507 08:06:33.625339 12681 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.617115 10186 raft_consensus.cc:1226] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.598534 13156 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.600167 12926 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:33.606189 13059 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:33.639573 12681 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:33.604941 13118 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:33.637888 10231 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.609532 13162 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.647606 10906 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.655655 10906 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.649937 10231 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.642647 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.660408 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.660545 10231 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.612221 13158 raft_consensus.cc:3032] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:33.661881 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.607385 12959 consensus_queue.cc:1046] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20240507 08:06:33.633976 13119 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:33.632408 12980 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:33.648692 10191 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.646545 13121 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:33.655705 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:33.668923 10943 raft_consensus.cc:2379] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 7: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:33.663120 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.599133 13159 consensus_queue.cc:1046] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20240507 08:06:33.655166 13118 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:33.665347 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:33.667011 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 125200us
I20240507 08:06:33.654381 12926 raft_consensus.cc:686] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:33.662237 10231 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.656919 13014 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:33.665082 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.679931 13156 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.682049 13156 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.690447 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.646761 13019 consensus_queue.cc:1035] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20240507 08:06:33.659580 13102 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.658731 12981 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:33.670789 12980 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:33.764568 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 102431us
W20240507 08:06:33.765758 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 90956.7us
I20240507 08:06:33.642190 13112 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20240507 08:06:33.779299 12980 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.780483 13112 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:33.782280 10187 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:33.783799 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:33.784408 10340 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:33.735718 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.664654 13158 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:33.657414 13111 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:33.668776 13161 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:33.752828 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.671742 13162 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:33.668501 13042 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:33.765547 10231 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.671018 13119 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.779176 13102 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:33.757153 13171 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.087s user 0.004s sys 0.000s
I20240507 08:06:33.752446 12986 raft_consensus.cc:3032] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:33.791898 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2113) took 1689 ms. Trace:
I20240507 08:06:33.792024 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.808286 10190 rpcz_store.cc:276] 0507 08:06:32.102161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.694308 (+592147us) service_pool.cc:225] Handling call
0507 08:06:32.694568 (+ 260us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:33.447815 (+753247us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:33.791692 (+343877us) spinlock_profiling.cc:243] Waited 753 ms on lock 0x7b54001f04d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:33.791843 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1731776896}
W20240507 08:06:33.758739 13122 log.cc:924] Time spent T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.086s user 0.000s sys 0.013s
I20240507 08:06:33.803033 10304 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.777774 13156 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:33.806087 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.806560 10232 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:33.810871 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.751228 12998 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240507 08:06:33.819829 12998 raft_consensus.cc:3032] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:33.820164 12998 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
W20240507 08:06:33.821921 10188 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2083) took 3048 ms (client timeout 1697 ms). Trace:
W20240507 08:06:33.822345 10188 rpcz_store.cc:269] 0507 08:06:30.773666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.870385 (+ 96719us) service_pool.cc:225] Handling call
0507 08:06:33.821715 (+2951330us) spinlock_profiling.cc:243] Waited 2.34 s on lock 0x7b5400260554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:33.821830 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5375131904}
I20240507 08:06:33.822719 12980 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:33.794075 12867 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240507 08:06:33.823531 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:33.824252 10277 tablet_copy_source_session.cc:205] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Tablet Copy: opened 0 blocks and 1 log segments
I20240507 08:06:33.824745 13169 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.825820 12981 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:33.824280 10264 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:33.828227 10261 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.828449 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:33.815649 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50234.7us
I20240507 08:06:33.829499 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.807986 12986 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:33.834375 10341 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:33.835590 10188 raft_consensus.cc:1261] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20240507 08:06:33.839577 10340 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:33.840706 10156 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:33.842223 13003 consensus_queue.cc:1035] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20240507 08:06:33.842804 13164 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20240507 08:06:33.843101 13119 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.840842 13168 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:33.760110 13142 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.843760 13168 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:33.844672 13168 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:33.845413 13142 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.816609 13112 consensus_queue.cc:1035] T 95da5fd11446429481eb60f5a0c8158e P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20240507 08:06:33.846192 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:33.793788 13161 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.834573 13111 raft_consensus.cc:686] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:33.847698 13158 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.791082 13121 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:33.856379 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.857278 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.883443 10190 raft_consensus.cc:2379] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 10: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:33.884034 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.826756 13042 raft_consensus.cc:686] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:33.846724 10266 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:33.803205 12952 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:33.760444 13125 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.863925 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.777371 13147 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:33.866859 13166 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
I20240507 08:06:33.820940 12998 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.863694 13132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c898839b7c84ce6ba0e135f778ad72d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:06:33.895838 13111 consensus_queue.cc:237] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.881397 12926 consensus_queue.cc:237] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.867166 12979 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.842047 12907 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.863786 12980 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:33.870342 13139 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:33.870261 10231 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.905614 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:33.871701 13168 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:33.871479 13118 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.909953 13121 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.911226 13118 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.873790 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2096) took 2701 ms. Trace:
I20240507 08:06:33.895876 12998 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:33.879069 10158 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:33.884513 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 81597.3us
I20240507 08:06:33.859510 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:33.841996 13141 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.755992 13059 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.863471 13117 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:33.890610 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.931113 12681 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:33.834321 13065 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:33.894369 13158 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:33.894619 12986 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.881577 13161 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:33.908535 12980 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.892035 12952 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:33.908701 10942 raft_consensus.cc:1226] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.904582 13139 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.911986 10183 rpcz_store.cc:276] 0507 08:06:31.172816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.250066 (+1077250us) service_pool.cc:225] Handling call
0507 08:06:32.250396 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:33.443892 (+1193496us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:33.873528 (+429636us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54001d9554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:33.873721 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2709440896}
I20240507 08:06:33.874508 13077 raft_consensus.cc:3032] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:33.892113 13102 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:33.929976 10264 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20240507 08:06:33.930284 13141 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.932025 10263 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.909006 13142 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:33.933943 10189 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:33.934391 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2109) took 1862 ms. Trace:
I20240507 08:06:33.961742 10186 rpcz_store.cc:276] 0507 08:06:32.071988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.631748 (+559760us) service_pool.cc:225] Handling call
0507 08:06:33.616959 (+985211us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:33.635213 (+ 18254us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:33.934216 (+299003us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b54003b3650. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:33.934337 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2132662528}
I20240507 08:06:33.909106 13119 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:33.963888 13119 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:33.964371 13119 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:33.938040 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:33.966599 12907 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:33.967098 10262 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 7
I20240507 08:06:33.939143 13065 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:33.939671 13063 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240507 08:06:33.940302 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 61395us
I20240507 08:06:33.940600 12986 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:33.941550 10190 raft_consensus.cc:1226] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20240507 08:06:33.863937 13089 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20240507 08:06:33.932940 10304 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:33.935812 13061 consensus_queue.cc:1046] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20240507 08:06:33.949379 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:34.027225 10341 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:34.027372 10185 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:33.907258 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:34.028545 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 121415us
I20240507 08:06:34.029227 10231 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:34.020959 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:33.955749 13077 raft_consensus.cc:729] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:33.955731 13139 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:34.024683 10262 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:34.038383 10906 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:34.039182 13177 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:34.039645 13177 raft_consensus.cc:686] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:34.040434 13177 consensus_queue.cc:237] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.041102 13077 consensus_queue.cc:260] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.050168 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.961053 13142 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:34.104626 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:34.107509 10232 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:34.112159 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 48952.9us
I20240507 08:06:34.122223 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:34.124095 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:33.962826 13102 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:34.135093 10189 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:34.135560 10341 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:34.107340 13141 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:34.137101 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:34.137459 10333 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:34.018710 13092 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.106s
I20240507 08:06:33.968953 13185 consensus_queue.cc:1046] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.175s
I20240507 08:06:33.953084 13042 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.114691 13142 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:34.109143 13156 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240507 08:06:34.108069 13169 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:34.119385 13159 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:34.172600 13095 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:34.120862 13147 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.210155 13195 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:34.142958 12979 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:34.220937 13183 log.cc:924] Time spent T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
W20240507 08:06:34.222383 13187 log.cc:924] Time spent T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
I20240507 08:06:34.182240 13125 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:34.212213 13193 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.066s user 0.002s sys 0.000s
I20240507 08:06:34.231698 13086 consensus_queue.cc:260] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:34.228798 13190 log.cc:924] Time spent T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.179s user 0.000s sys 0.001s
I20240507 08:06:34.234524 13102 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.189867 13191 consensus_queue.cc:1046] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20240507 08:06:34.235771 13191 raft_consensus.cc:3032] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:34.236117 13191 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:34.236903 13191 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.229923 13139 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.254469 13139 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.254995 13142 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.258929 13142 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.259209 13119 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.260673 13119 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.295619 10186 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
W20240507 08:06:34.178056 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 245283us
I20240507 08:06:34.297261 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:34.175627 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 164106us
I20240507 08:06:34.310294 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.312047 10333 raft_consensus.cc:2379] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:34.339206 13102 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.339725 13152 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:34.340291 13102 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.343072 13200 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.344081 13200 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.347215 13203 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.348300 13203 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:34.340672 13120 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.004s sys 0.004s
W20240507 08:06:34.350311 13120 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.655s user 0.004s sys 0.005s
I20240507 08:06:34.217451 13095 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.120678 12980 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.354609 13095 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.358222 12952 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.359007 13102 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:34.363035 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 994424us
I20240507 08:06:34.349781 13149 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:34.450219 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 97057.3us
W20240507 08:06:34.462222 13172 log.cc:924] Time spent T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.784s user 0.005s sys 0.006s
I20240507 08:06:34.465718 10304 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.466751 13092 raft_consensus.cc:3032] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
W20240507 08:06:34.467293 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 142667us
I20240507 08:06:34.468622 10304 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.469303 13059 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:34.470083 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.471403 13149 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.478561 12807 raft_consensus.cc:3032] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:34.479022 12807 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:34.479652 13141 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:34.480301 12807 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.493978 10155 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:34.495198 13161 consensus_meta.cc:287] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.505s user 0.006s sys 0.006s
I20240507 08:06:34.495848 13203 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.495638 13161 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.498036 10258 raft_consensus.cc:2379] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 9: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:34.511812 13149 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.467268 13092 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:34.515537 13092 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.518265 13161 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.517774 13092 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:34.519698 13177 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.520390 13177 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.522667 13177 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.524787 13086 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:34.528369 13189 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:34.529840 13189 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20240507 08:06:34.531419 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 63007.2us
I20240507 08:06:34.532610 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.533977 13185 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:34.535789 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:34.477456 13117 consensus_meta.cc:287] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.544s user 0.001s sys 0.017s
I20240507 08:06:34.537230 13117 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20240507 08:06:34.537966 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 81455.4us
W20240507 08:06:34.541689 12998 consensus_meta.cc:287] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.612s user 0.008s sys 0.009s
I20240507 08:06:34.541774 13111 consensus_queue.cc:1046] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:34.542340 13200 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.547823 10943 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:34.551098 13019 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.551801 13019 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.555054 13077 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:34.558367 10939 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 213) took 2036 ms. Trace:
W20240507 08:06:34.527043 10184 consensus_meta.cc:287] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.698s user 0.015s sys 0.002s
I20240507 08:06:34.559237 10184 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:34.558943 10939 rpcz_store.cc:276] 0507 08:06:32.521665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.521802 (+ 137us) service_pool.cc:225] Handling call
0507 08:06:32.890797 (+368995us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b58002f2e98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec56dc 00007fcf8dec529b 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:06:33.374606 (+483809us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:33.393245 (+ 18639us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:34.557973 (+1164728us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54007f03d0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:34.558168 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1156085760}
I20240507 08:06:34.560580 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2127) took 1027 ms. Trace:
I20240507 08:06:34.560926 10184 rpcz_store.cc:276] 0507 08:06:33.533014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:33.533135 (+ 121us) service_pool.cc:225] Handling call
0507 08:06:34.560369 (+1027234us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400046d54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:34.560496 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":628811136}
I20240507 08:06:34.558454 13053 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:34.561790 13053 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:34.562942 13111 consensus_queue.cc:1225] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240507 08:06:34.563699 10942 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:34.564601 13111 raft_consensus.cc:3032] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:34.565145 13111 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:34.567813 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.567860 13111 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.566293 13061 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:34.568921 13203 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:34.571388 12979 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:34.576998 13061 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:34.573184 10266 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:34.575500 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:34.563133 13053 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.581945 10261 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:34.582466 13053 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:34.570365 10304 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:34.586794 13111 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:34.587499 13061 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:34.587965 13061 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:34.588397 13061 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:34.611029 13169 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:34.606256 12807 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:34.635885 10934 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43910 (request call id 244) took 2175 ms (client timeout 2166 ms). Trace:
W20240507 08:06:34.636461 10934 rpcz_store.cc:269] 0507 08:06:32.460171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.461111 (+ 940us) service_pool.cc:225] Handling call
0507 08:06:34.635397 (+2174286us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b5400410354. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:34.635639 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2282545920}
I20240507 08:06:34.637529 13147 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Stepping down as leader of term 9
I20240507 08:06:34.637758 10906 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:34.638125 10935 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:34.638839 13204 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:34.639230 13204 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:34.639894 13204 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:34.640117 13019 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:34.644922 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 206760us
I20240507 08:06:34.639218 13049 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:34.638149 13147 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:34.647328 10155 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.647413 13147 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.648864 10155 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.648957 13204 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.629429 13115 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20240507 08:06:34.649138 13147 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:34.654436 10233 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:34.640223 10935 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 212) took 2118 ms. Trace:
W20240507 08:06:34.655009 13173 log.cc:924] Time spent T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.976s user 0.017s sys 0.000s
I20240507 08:06:34.656324 13152 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:34.668375 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 98237.3us
I20240507 08:06:34.669490 12976 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20240507 08:06:34.663342 12681 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.677047 13049 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:34.694164 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.681402 13169 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:34.686666 13086 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.656641 10935 rpcz_store.cc:276] 0507 08:06:32.521370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.568196 (+ 46826us) service_pool.cc:225] Handling call
0507 08:06:32.892718 (+324522us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b58002f2e98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:06:34.637744 (+1745026us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:34.638526 (+ 782us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:34.639993 (+ 1467us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b5400410350. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:34.640138 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4038923904}
W20240507 08:06:34.700399 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 46295.2us
I20240507 08:06:34.692685 12681 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:34.706310 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 57643.9us
W20240507 08:06:34.697679 13121 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s user 0.031s sys 0.097s
I20240507 08:06:34.714056 10943 raft_consensus.cc:1226] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:34.694347 13204 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.718482 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:34.722633 10266 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:34.723277 13086 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.725349 10187 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:34.726797 13086 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.729861 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.730583 10338 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:34.731487 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:34.699888 13152 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.732513 10337 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:34.732668 10335 raft_consensus.cc:1261] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:34.733767 13152 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.733968 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:34.734402 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:34.734503 13152 consensus_queue.cc:1046] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:34.735119 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.735783 10189 raft_consensus.cc:1261] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:34.735961 13152 raft_consensus.cc:3032] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:34.736368 13152 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:34.737867 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:34.719208 13169 consensus_queue.cc:1046] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20240507 08:06:34.738660 10185 raft_consensus.cc:2379] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:34.739054 13169 raft_consensus.cc:3032] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:34.739415 13169 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:34.740396 13169 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.746862 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.747639 10340 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:34.720211 13049 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.749737 10190 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:34.750191 13049 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.751250 10156 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:34.751463 10259 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:34.758075 12681 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.772454 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:34.774268 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 54810.3us
I20240507 08:06:34.733841 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.775936 13132 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:43910 (request call id 246) took 2280 ms. Trace:
I20240507 08:06:34.776077 13169 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:34.776381 13132 rpcz_store.cc:276] 0507 08:06:32.474961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.475335 (+ 374us) service_pool.cc:225] Handling call
0507 08:06:32.892162 (+416827us) ts_tablet_manager.cc:903] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
0507 08:06:33.846245 (+954083us) tablet_copy_client.cc:361] Tablet copy session begun
0507 08:06:34.388841 (+542596us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:34.454559 (+ 65718us) tablet_copy_client.cc:490] Wrote new tablet metadata
0507 08:06:34.742128 (+287569us) tablet_copy_client.cc:411] Wrote new consensus metadata
0507 08:06:34.742568 (+ 440us) tablet_replica.cc:205] Creating consensus instance
0507 08:06:34.742940 (+ 372us) ts_tablet_manager.cc:958] Replica registered.
0507 08:06:34.743050 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":237,"tablet-copy.queue_time_us":392125,"thread_start_us":374727,"threads_started":2}
I20240507 08:06:34.777870 13132 tablet_copy_client.cc:791] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 0 data blocks...
I20240507 08:06:34.778481 13132 tablet_copy_client.cc:655] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: tablet copy: Starting download of 1 WAL segments...
I20240507 08:06:34.781888 13019 consensus_queue.cc:1035] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:34.783085 13083 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:34.783484 13083 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:34.783828 13083 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:34.785734 13120 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:34.803328 13157 consensus_queue.cc:1046] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:34.806159 10266 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:34.765138 13049 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.807982 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:34.814723 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:34.814838 12980 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:34.815492 12980 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:34.815863 12980 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:34.818022 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:34.737430 13152 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.885980 10189 raft_consensus.cc:2379] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 7: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:34.886842 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.887641 10339 raft_consensus.cc:3032] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:34.810245 13077 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240507 08:06:34.817273 13195 consensus_queue.cc:1046] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240507 08:06:34.814941 13205 raft_consensus.cc:3032] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
W20240507 08:06:34.893138 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 70252.3us
I20240507 08:06:34.894984 10339 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:34.896037 10306 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:34.892220 13125 consensus_queue.cc:1035] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20240507 08:06:34.902583 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:34.903398 10188 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:34.904415 13152 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:34.898404 13095 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.901775 10906 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:34.914052 13151 consensus_queue.cc:1046] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20240507 08:06:34.915432 13168 consensus_queue.cc:1046] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:34.916361 13168 consensus_queue.cc:1225] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer 59382e6647c549d89d83c70233c0667e log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240507 08:06:34.916885 13168 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 LEADER]: Stepping down as leader of term 9
I20240507 08:06:34.913707 13095 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.917256 13168 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:34.921172 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 87238.5us
I20240507 08:06:34.921893 13168 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.895802 13117 consensus_queue.cc:1046] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20240507 08:06:34.923226 13168 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:34.923130 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:34.923408 13117 raft_consensus.cc:3032] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:34.924005 13117 raft_consensus.cc:729] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:34.925930 12867 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:34.926605 12867 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.899116 13205 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:34.930204 12867 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.932618 13205 consensus_queue.cc:260] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.933811 13095 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.934955 13205 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:34.925076 13117 consensus_queue.cc:260] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.937541 10934 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:34.942340 13042 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:34.967434 13149 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:34.967964 13042 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.945446 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:34.970315 10263 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:34.971367 13042 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:34.945907 13117 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:34.952015 13063 raft_consensus.cc:3032] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:34.973980 13063 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:34.974988 13063 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.976087 13063 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:34.952243 12807 raft_consensus.cc:3032] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:34.977823 12807 raft_consensus.cc:729] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:34.977896 10906 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:34.978792 13042 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:34.979295 13042 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:34.978842 12807 consensus_queue.cc:260] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.979640 13042 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:34.980640 12807 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:34.957530 13086 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.982380 13086 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:34.985070 13086 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:34.985522 13204 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:34.942970 13132 tablet_copy_client.cc:531] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240507 08:06:34.943143 13177 raft_consensus.cc:2781] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:34.986176 13204 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:34.958091 13191 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:35.027827 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.015913 10339 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.032701 10908 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:35.028026 10340 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:35.034895 10339 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.035043 13095 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:35.035589 13095 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:35.035840 13147 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:35.036064 13095 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.039490 13083 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.044874 13204 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.045433 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.045789 10184 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:35.046188 10263 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
W20240507 08:06:35.045562 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 152846us
I20240507 08:06:35.048522 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:35.050230 13198 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:35.050391 13147 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:35.050992 10906 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:35.051784 13115 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:35.051607 13147 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.052212 13115 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.052261 10155 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:35.052649 13115 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.053468 13053 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:35.050182 12731 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:35.010010 13177 raft_consensus.cc:686] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [term 8 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:35.056535 10908 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:35.060873 10341 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20240507 08:06:35.063146 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 169644us
I20240507 08:06:35.069494 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.073535 10340 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:35.041500 13083 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:35.081913 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 195278us
I20240507 08:06:35.063252 13053 raft_consensus.cc:686] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:35.079833 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.094537 10186 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:35.094758 10333 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:35.007764 12980 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.015857 13086 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:35.096324 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.013517 13061 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.109617 10089 catalog_manager.cc:5484] T 7a181f4ae4d342129ad262e5545debc9 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 7 to 8, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 8 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:35.111093 13053 consensus_queue.cc:237] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.111550 10089 catalog_manager.cc:5484] T beb5387c55024337b72bb8aa1ba09a7b P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 5 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 6 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240507 08:06:35.113791 13221 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.177s user 0.003s sys 0.000s
I20240507 08:06:35.114238 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.115037 10184 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:35.115785 12731 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:35.117223 13141 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.121786 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.122417 13141 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.123656 12786 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.124717 13141 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.125098 10264 raft_consensus.cc:1261] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:35.118065 13199 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.126221 13199 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.127493 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.128140 10260 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:35.128197 13199 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.118410 13177 consensus_queue.cc:237] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.113353 13083 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:35.120478 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.063418 13169 consensus_queue.cc:1046] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20240507 08:06:35.132190 10155 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:35.113189 10089 catalog_manager.cc:5484] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 6 to 7, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 7 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:35.130076 10267 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:35.122519 10259 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.140805 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.109663 12980 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.142154 10338 raft_consensus.cc:3037] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:35.143206 10340 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:35.145413 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.146199 10333 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:35.144093 12998 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.150830 10156 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:35.156965 13192 log.cc:924] Time spent T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20240507 08:06:35.158080 13223 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240507 08:06:35.153067 12998 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.160017 13047 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.160295 12998 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.160516 13047 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.161075 13077 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:35.161698 13077 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:35.162564 13077 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.170099 10267 raft_consensus.cc:1226] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:35.170331 13111 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:35.171066 10087 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59948 (request call id 219) took 3173 ms. Trace:
I20240507 08:06:35.170794 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.172253 10262 raft_consensus.cc:2396] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20240507 08:06:35.173977 10258 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:35.174650 10089 catalog_manager.cc:5484] T bcbe630c1736458898143fb230a9ffb8 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 8 to 9, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 9 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:35.178292 10089 catalog_manager.cc:5484] T 84da12b367b046b7bcfb08aaa6e1c97b P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 8 to 9, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 9 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:35.180855 10338 raft_consensus.cc:2454] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:35.137142 13083 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.155700 12980 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:35.147845 13086 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.194074 10089 catalog_manager.cc:5484] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 6 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:35.162683 13047 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.203899 10089 catalog_manager.cc:5484] T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:35.205844 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.206537 10260 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:35.213753 13086 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.217741 13083 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.221752 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.222615 10265 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:35.217783 13061 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:35.229760 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.232816 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.233309 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.236394 10339 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:35.179981 13169 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:35.241133 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.243266 10333 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
W20240507 08:06:35.247181 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 146433us
I20240507 08:06:35.170895 13111 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.154983 13095 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.225559 12786 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:35.256989 13083 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.259900 10156 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:35.267330 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.276434 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.277673 10185 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:35.277074 10260 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 4.
I20240507 08:06:35.276646 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:35.260735 12980 raft_consensus.cc:686] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:35.301330 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.171461 10087 rpcz_store.cc:276] 0507 08:06:31.981484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:31.981681 (+ 197us) service_pool.cc:225] Handling call
0507 08:06:35.151122 (+3169441us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:33.694668 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:33.867335 (+172667us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:33.867375 (+ 40us) write_op.cc:435] Acquired schema lock
0507 08:06:33.867400 (+ 25us) tablet.cc:660] Decoding operations
0507 08:06:33.868207 (+ 807us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:33.868297 (+ 90us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:33.868328 (+ 31us) tablet.cc:683] Acquiring locks for 6 operations
0507 08:06:33.868872 (+ 544us) tablet.cc:699] Row locks acquired
0507 08:06:33.868891 (+ 19us) write_op.cc:260] PREPARE: finished
0507 08:06:33.869038 (+ 147us) write_op.cc:270] Start()
0507 08:06:33.869153 (+ 115us) write_op.cc:276] Timestamp: P: 1715069193868975 usec, L: 0
0507 08:06:33.869190 (+ 37us) op_driver.cc:331] REPLICATION: starting
0507 08:06:33.914355 (+ 45165us) log.cc:842] Serialized 2472 byte log entry
0507 08:06:34.257543 (+343188us) op_driver.cc:444] REPLICATION: finished
0507 08:06:34.610086 (+352543us) write_op.cc:301] APPLY: starting
0507 08:06:34.610234 (+ 148us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:34.610541 (+ 307us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:34.610715 (+ 174us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:34.611761 (+ 1046us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:34.619212 (+ 7451us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0507 08:06:34.619285 (+ 73us) write_op.cc:312] APPLY: finished
0507 08:06:34.622547 (+ 3262us) log.cc:842] Serialized 53 byte log entry
0507 08:06:34.627347 (+ 4800us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:34.630959 (+ 3612us) write_op.cc:454] Released schema lock
0507 08:06:34.631051 (+ 92us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:34.631517 (+ 466us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:34.631559 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":332017,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":352094,"apply.run_cpu_time_us":4341,"apply.run_wall_time_us":22003,"num_ops":6,"prepare.queue_time_us":231523,"prepare.run_cpu_time_us":17760,"prepare.run_wall_time_us":301200,"raft.queue_time_us":25685,"raft.run_cpu_time_us":67936,"raft.run_wall_time_us":181721,"replication_time_us":388265,"thread_start_us":350627,"threads_started":4,"wal-append.queue_time_us":206549}]]}
I20240507 08:06:35.279182 12731 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:35.308331 10906 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:35.280526 13141 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:35.262107 13169 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.320127 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.285287 13092 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.277365 13083 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.326661 10265 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:35.332391 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.321717 12731 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:35.338764 12731 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.313927 13120 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:35.350193 10187 raft_consensus.cc:1261] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:35.347558 10259 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:35.319190 13141 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:35.364493 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.322507 12807 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.366585 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.367671 10265 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:35.380735 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.384133 10262 raft_consensus.cc:1226] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:35.392238 10260 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:35.317937 13121 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.349962 13120 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.393859 10183 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:35.366997 13198 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:35.356842 13092 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.405207 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.407668 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.406571 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.408407 10189 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 9.
I20240507 08:06:35.347173 13042 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.403081 13120 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:35.430621 10185 raft_consensus.cc:2379] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:35.431829 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.434275 10259 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:35.434301 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.436241 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.394229 13118 consensus_queue.cc:1035] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.188s
I20240507 08:06:35.402812 12807 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.447588 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.405823 13117 consensus_queue.cc:1035] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20240507 08:06:35.414265 13142 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:35.470194 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.472863 13169 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.474637 10340 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
W20240507 08:06:35.476848 13226 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.089s user 0.001s sys 0.001s
I20240507 08:06:35.442579 13095 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.476145 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.439224 13111 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.460361 10333 raft_consensus.cc:1226] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:35.478911 10265 raft_consensus.cc:2379] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 10: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:35.486327 10908 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:35.481961 13229 log.cc:924] Time spent T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20240507 08:06:35.426297 12980 consensus_queue.cc:237] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.493196 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:35.500820 10183 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.494028 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.492960 10185 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 3.
I20240507 08:06:35.511154 10189 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:35.513203 10306 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:35.514451 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.515487 10259 raft_consensus.cc:2396] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20240507 08:06:35.430666 13141 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.519709 10333 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:35.492942 13204 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:35.505129 13111 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.499971 10934 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:35.524168 10259 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20240507 08:06:35.530992 13233 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240507 08:06:35.533128 10263 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: []
W20240507 08:06:35.534067 13230 log.cc:924] Time spent T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
W20240507 08:06:35.534245 13232 log.cc:924] Time spent T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:06:35.534583 13083 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:35.522480 12867 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:35.536672 12867 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.537024 12867 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:35.539032 10265 raft_consensus.cc:1261] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240507 08:06:35.473274 13121 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.543941 10333 raft_consensus.cc:1203] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 8.5->[8.6-8.6] Dedup: 8.6->[]
I20240507 08:06:35.549274 10333 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20240507 08:06:35.539377 10265 pending_rounds.cc:76] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 5
I20240507 08:06:35.550231 10265 pending_rounds.cc:97] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 8.6
I20240507 08:06:35.557739 10265 raft_consensus.cc:2894] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Aborting config change with OpId 8.6: Aborted: Op aborted by new leader
I20240507 08:06:35.564713 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.568373 10259 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:35.552783 13141 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:35.576989 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.578788 10265 raft_consensus.cc:2866] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:35.579797 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.580610 10259 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:35.581624 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.583094 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.571015 10260 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
W20240507 08:06:35.583276 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 92074.2us
I20240507 08:06:35.522430 13142 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.585044 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.582370 10266 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:35.561596 13092 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.546219 13111 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.585368 10261 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:35.594333 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.595119 10332 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 4.
I20240507 08:06:35.596030 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:35.596334 10231 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.131:37047 timed out after 1.542s (SENT) [suppressed 43 similar messages]
I20240507 08:06:35.597983 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.596374 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.600819 13198 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:35.601445 10266 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 10
I20240507 08:06:35.602054 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.605510 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:35.605947 13205 consensus_meta.cc:287] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.670s user 0.003s sys 0.008s
W20240507 08:06:35.606451 13235 log.cc:924] Time spent T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.129s user 0.002s sys 0.000s
W20240507 08:06:35.608853 13115 consensus_meta.cc:287] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.556s user 0.011s sys 0.000s
I20240507 08:06:35.610208 13111 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.593643 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:39102 (request call id 7) took 5991 ms. Trace:
I20240507 08:06:35.610754 10265 rpcz_store.cc:276] 0507 08:06:29.604123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:30.476429 (+872306us) service_pool.cc:225] Handling call
0507 08:06:30.720744 (+244315us) log.cc:842] Serialized 506 byte log entry
0507 08:06:35.593500 (+4872756us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3111,"raft.queue_time_us":3412,"raft.run_cpu_time_us":1449,"raft.run_wall_time_us":1442,"spinlock_wait_cycles":104704,"thread_start_us":8801,"threads_started":1,"wal-append.queue_time_us":185385,"wal-append.run_cpu_time_us":2474,"wal-append.run_wall_time_us":1254223}
I20240507 08:06:35.612964 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.594127 10337 raft_consensus.cc:1226] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:35.614504 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.614668 13121 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.615430 13132 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Bootstrap starting.
I20240507 08:06:35.615581 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.616636 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.594440 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:35.617969 10189 raft_consensus.cc:3037] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:35.620461 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.621024 10263 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:35.593755 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.622119 10335 raft_consensus.cc:2379] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 8: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:35.623778 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.624058 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.624411 10190 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:35.624655 10185 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:35.624888 10259 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:35.626479 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.627480 10232 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:35.627601 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.628310 10259 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:35.628469 10232 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:35.629595 13169 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:35.629717 13121 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:35.630056 13169 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.630174 13121 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.630584 13121 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:35.630514 13169 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:35.637813 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.638521 10337 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 2.
I20240507 08:06:35.640743 10266 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 10.
I20240507 08:06:35.642025 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.639849 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:35.585309 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.644028 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.644934 10266 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 2.
I20240507 08:06:35.629217 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.649627 10340 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
W20240507 08:06:35.651825 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 66595us
I20240507 08:06:35.555276 13047 consensus_queue.cc:1046] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
I20240507 08:06:35.658032 10906 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:35.654289 10259 raft_consensus.cc:2354] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8.
I20240507 08:06:35.655107 10263 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:35.568739 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:35.660701 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:35.656632 10260 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:35.661159 10189 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 10.
I20240507 08:06:35.653856 10334 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:35.658674 10267 raft_consensus.cc:1226] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:35.657218 10183 raft_consensus.cc:2454] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:35.659317 10258 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:35.663744 10908 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:35.655017 10262 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:35.643692 10335 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:35.670315 10261 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:35.583771 13042 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.674660 10341 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:35.676584 10259 raft_consensus.cc:1226] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:35.681542 10262 raft_consensus.cc:1226] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:35.683362 10334 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:35.609295 13115 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.689496 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:35.692250 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 96118.9us
I20240507 08:06:35.694046 10266 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:35.670027 13111 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:35.701897 10267 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240507 08:06:35.673691 13092 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:35.677664 12807 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.688097 13061 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:35.654027 13169 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.705396 10934 raft_consensus.cc:2379] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 10: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:35.717274 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:35.737176 10258 raft_consensus.cc:1226] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:35.709957 13177 raft_consensus.cc:2781] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:35.700292 13049 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20240507 08:06:35.738139 12681 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
W20240507 08:06:35.745800 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 51490.4us
I20240507 08:06:35.754585 10304 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:35.749630 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 97349.8us
I20240507 08:06:35.726624 13147 consensus_queue.cc:1035] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
W20240507 08:06:35.770958 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 117205us
I20240507 08:06:35.750383 10336 raft_consensus.cc:1261] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20240507 08:06:35.752257 10333 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:35.762861 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:35.773470 10304 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:35.770169 13142 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.774883 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:35.768112 13241 log.cc:924] Time spent T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240507 08:06:35.713979 13061 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.729475 13042 raft_consensus.cc:2781] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 10 FOLLOWER]: Leader election won for term 10
W20240507 08:06:35.779399 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 115774us
I20240507 08:06:35.710284 13092 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.710491 13111 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:35.799338 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.804026 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.805665 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.808655 12681 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:35.809089 10332 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:35.809129 12681 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.809803 12681 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:35.774102 12807 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:35.814677 10190 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240507 08:06:35.815259 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:35.816123 13061 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.821696 13111 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:35.824899 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.829052 12681 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:35.834101 13237 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
I20240507 08:06:35.797188 13047 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:35.806244 13142 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:35.913017 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 136891us
I20240507 08:06:35.914320 12681 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:35.913971 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:35.921761 10336 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240507 08:06:35.934185 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:35.933859 13142 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.936074 13142 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:35.937304 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:35.938916 10304 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:35.939944 10304 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:35.842774 12807 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:35.937317 12681 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:35.802242 13191 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20240507 08:06:35.946619 12681 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:35.941901 13120 consensus_meta.cc:287] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.500s user 0.006s sys 0.003s
I20240507 08:06:35.967671 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:35.940665 13125 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:35.925467 13061 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:35.943130 12807 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:35.992055 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:35.951392 13047 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:35.998477 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:35.995433 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 222648us
W20240507 08:06:35.997540 10138 auto_rebalancer.cc:254] failed to send replica move request: Aborted: Op aborted by new leader
I20240507 08:06:36.000448 13092 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.000890 10261 raft_consensus.cc:2379] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 10: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:36.001046 13092 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:36.001477 13092 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.001847 13092 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:36.005793 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 190711us
I20240507 08:06:36.006130 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.006592 10231 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.009654 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:36.011287 13249 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20240507 08:06:36.012650 13169 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.014729 13254 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.015215 13254 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:35.980876 13255 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:36.017252 13255 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.023134 13255 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.023602 12867 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.017899 13198 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:35.980767 13061 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:36.018249 10264 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:36.018348 13199 raft_consensus.cc:3032] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:36.028036 13061 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:35.998513 13047 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.028573 12867 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.028848 13047 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:35.969563 13142 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.031080 13142 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.027944 13198 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:36.032853 13198 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:36.032970 13142 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:36.028941 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.034904 10231 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.035692 13141 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.036319 13141 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:36.036671 13141 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.036999 13141 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:36.019506 13117 consensus_queue.cc:1046] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:35.955639 13115 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.038445 13111 raft_consensus.cc:3032] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:36.038836 13111 raft_consensus.cc:729] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:36.039052 10260 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:36.039660 13111 consensus_queue.cc:260] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.040266 10265 raft_consensus.cc:2396] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20240507 08:06:36.040915 13111 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
W20240507 08:06:36.000466 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 60621.3us
I20240507 08:06:35.961829 13177 raft_consensus.cc:686] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:36.042124 10304 leader_election.cc:398] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8.
I20240507 08:06:36.042407 10304 leader_election.cc:401] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20240507 08:06:36.042733 13177 consensus_queue.cc:237] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.043764 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:36.044986 12867 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8.
I20240507 08:06:35.962074 13253 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20240507 08:06:36.045753 12867 consensus_queue.cc:1046] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:36.047837 13256 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:36.048272 13256 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:35.975381 13120 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.048635 13256 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.049904 13271 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:36.051369 13271 consensus_queue.cc:1046] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:36.052223 13271 raft_consensus.cc:3032] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:36.054281 13271 raft_consensus.cc:729] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.054461 10233 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:36.055145 13271 consensus_queue.cc:260] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.055720 10260 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240507 08:06:36.028384 13199 raft_consensus.cc:729] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.056797 13254 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.057327 13199 consensus_queue.cc:260] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.058904 13199 raft_consensus.cc:3037] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.016199 13121 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.050185 13120 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.056295 13271 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.062454 13121 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:36.067245 13244 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.353s user 0.000s sys 0.002s
I20240507 08:06:36.070432 13169 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:36.072502 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:36.075177 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.051671 12867 raft_consensus.cc:3032] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:36.082659 12681 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.090098 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.091678 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:36.092458 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.093148 10258 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.091939 10263 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240507 08:06:36.096464 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.091989 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:36.098014 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.100801 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.101730 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.101797 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.102155 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.102507 10260 raft_consensus.cc:2379] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 6: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:36.102731 10186 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:36.103046 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.103397 10264 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:36.103637 10184 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 5
I20240507 08:06:36.103946 12681 consensus_queue.cc:1046] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240507 08:06:36.096454 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.106063 10267 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
W20240507 08:06:36.109238 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 83774.3us
I20240507 08:06:36.111673 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.112454 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.112687 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.113066 10183 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:36.113188 10189 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.117942 13277 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:36.118440 13277 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.121559 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.123114 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.099452 10306 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.127956 10267 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:36.122278 10192 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:36.134101 10304 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:36.135285 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 43977.9us
I20240507 08:06:36.085289 12867 raft_consensus.cc:729] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.135872 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.139878 10263 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
W20240507 08:06:36.135488 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 60425.6us
I20240507 08:06:36.137857 10188 raft_consensus.cc:1226] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20240507 08:06:36.104707 13063 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.107343 13169 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.141649 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.143067 10306 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.146705 12681 raft_consensus.cc:3032] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:36.147192 12681 raft_consensus.cc:729] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.132481 13245 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20240507 08:06:36.152734 10262 raft_consensus.cc:1226] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:36.155624 10258 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:36.140919 12731 consensus_queue.cc:1046] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240507 08:06:36.136974 13042 raft_consensus.cc:686] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [term 10 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:36.132933 13168 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.157445 12681 consensus_queue.cc:260] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.159511 10908 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:36.154474 10184 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:36.153066 10260 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
I20240507 08:06:36.164067 10189 raft_consensus.cc:2454] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:36.199026 10191 raft_consensus.cc:1226] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240507 08:06:36.138401 13086 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.190904 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.204993 10188 raft_consensus.cc:2379] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
W20240507 08:06:36.205281 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 45861.4us
W20240507 08:06:36.200790 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 62269.9us
I20240507 08:06:36.206737 10908 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:36.203209 10332 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:36.210363 10264 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:36.213382 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.214781 12681 raft_consensus.cc:3037] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.217976 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.218595 10264 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:36.219642 10906 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:36.219880 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 63120us
I20240507 08:06:36.221841 10304 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.223073 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.224237 10304 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:36.242666 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 99727.9us
I20240507 08:06:36.243643 10306 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.248934 13285 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.249662 13270 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:36.250128 13270 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:36.250512 13270 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:36.150801 13253 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20240507 08:06:36.218360 13226 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20240507 08:06:36.255147 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 85062.2us
I20240507 08:06:36.213136 13191 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:36.152336 13063 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.213243 13019 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:36.218418 13277 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.255592 13269 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:36.210736 13047 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.262234 13019 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.252032 13083 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20240507 08:06:36.246524 12867 consensus_queue.cc:260] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.282370 12681 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:36.280681 10232 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:36.265311 13282 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:36.276118 13191 raft_consensus.cc:686] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.287864 13278 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader election won for term 5
I20240507 08:06:36.288503 13278 raft_consensus.cc:686] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:36.288605 13191 consensus_queue.cc:237] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.289464 13278 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.280424 13042 consensus_queue.cc:237] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.298812 13280 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:36.299512 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 93137.3us
I20240507 08:06:36.265836 13256 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.300256 10155 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.301029 13152 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.301507 13256 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.301679 13152 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:36.301970 10155 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.302278 13083 consensus_queue.cc:1046] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:36.302888 13152 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:36.303272 13152 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:36.303833 13152 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.304131 12731 consensus_queue.cc:1046] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:36.305008 13205 raft_consensus.cc:3032] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:36.305339 13205 raft_consensus.cc:729] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.306064 13205 consensus_queue.cc:260] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.306982 13205 raft_consensus.cc:3037] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.308586 10186 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:36.303306 13083 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:36.309528 13083 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.309708 13274 raft_consensus.cc:2781] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:36.310150 13063 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.310462 13274 raft_consensus.cc:686] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.311537 13274 consensus_queue.cc:237] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.280453 13053 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240507 08:06:36.265564 12807 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.313155 13083 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.314543 13083 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:36.315258 12807 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.281064 13019 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.299314 13280 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.303468 10155 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:36.322722 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 49453.6us
I20240507 08:06:36.332835 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.341960 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.344694 10336 raft_consensus.cc:2396] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240507 08:06:36.329993 13157 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.316080 13061 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.354238 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:36.354429 13276 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:36.354904 10338 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.355026 13276 raft_consensus.cc:686] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.356113 13276 consensus_queue.cc:237] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.358521 13047 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.361616 13092 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.364172 13092 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.350535 13157 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.367465 13157 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:36.370419 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 67051.6us
I20240507 08:06:36.371105 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.371352 10155 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.374068 13275 raft_consensus.cc:3032] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:36.374522 13275 raft_consensus.cc:729] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.375558 13275 consensus_queue.cc:260] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.377103 13275 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
W20240507 08:06:36.379493 13261 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.341s user 0.002s sys 0.000s
I20240507 08:06:36.381493 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.330229 13198 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.413302 10341 raft_consensus.cc:1261] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240507 08:06:36.413440 10335 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:36.414491 10334 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:36.418394 10334 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:36.404747 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.407821 10338 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
W20240507 08:06:36.508383 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 228063us
W20240507 08:06:36.506544 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 355231us
I20240507 08:06:36.530119 10231 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.530588 10232 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:36.536278 10231 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:36.507717 10233 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 230448us
I20240507 08:06:36.522250 10186 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240507 08:06:36.428695 13053 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.502907 13019 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.458308 13270 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.426568 13062 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:36.444186 13280 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.443279 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.457144 13269 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20240507 08:06:36.459412 13266 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.422s user 0.003s sys 0.000s
I20240507 08:06:36.509339 13111 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:36.506898 12979 consensus_queue.cc:1046] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:36.349843 13141 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.513799 10337 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:36.429865 13152 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.444298 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.428963 13061 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.470882 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.423488 13157 consensus_queue.cc:1035] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20240507 08:06:36.437117 13198 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.551123 13121 raft_consensus.cc:2726] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:36.567648 10332 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:36.447003 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.551425 13168 raft_consensus.cc:2781] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
W20240507 08:06:36.568473 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 121650us
I20240507 08:06:36.572011 10155 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.552556 13062 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:36.572744 10190 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:36.573892 10306 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:36.569999 10339 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:36.570952 10332 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:36.560552 13111 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.583006 10190 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:36.584451 10155 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.585199 13083 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:36.585881 13083 raft_consensus.cc:686] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.586817 13157 consensus_queue.cc:1046] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:36.586789 13083 consensus_queue.cc:237] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.587437 12731 consensus_queue.cc:1046] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:36.587982 13157 raft_consensus.cc:3032] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:36.588121 13141 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.588357 13157 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.589030 12731 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:36.589682 13152 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.588429 13063 raft_consensus.cc:3032] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:36.578224 13121 raft_consensus.cc:2726] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:36.590574 13270 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.590584 13063 raft_consensus.cc:729] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:36.592384 13063 consensus_queue.cc:260] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.593628 13063 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.593768 13086 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240507 08:06:36.578058 13168 raft_consensus.cc:686] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:36.596035 13168 consensus_queue.cc:237] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.553463 13263 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20240507 08:06:36.600170 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:36.606722 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 382740us
I20240507 08:06:36.605906 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.574855 13276 raft_consensus.cc:2741] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240507 08:06:36.606954 10190 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.605545 13019 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.610335 10185 raft_consensus.cc:2379] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:36.613983 10155 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.576608 13198 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:36.576184 13275 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:36.608788 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.623705 10259 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.621879 10332 raft_consensus.cc:1261] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:36.573519 13062 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.620877 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.628672 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.629427 10258 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:36.599043 13111 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:36.635356 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 99222us
I20240507 08:06:36.577529 13205 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.039s
I20240507 08:06:36.636718 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.639423 10332 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:36.640202 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 86491.7us
I20240507 08:06:36.644629 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.644693 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.645352 10260 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.645546 10267 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 5: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:36.649282 10259 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:36.650549 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.630543 13282 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.652902 13282 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.653968 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.654984 13282 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.655738 13304 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:36.656234 13304 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.658315 13304 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:36.666440 13205 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:36.667104 13304 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:36.667266 13205 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.660048 10267 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240507 08:06:36.663450 10258 raft_consensus.cc:2454] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:36.632422 13276 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:36.663115 10264 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240507 08:06:36.669669 10267 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:36.669364 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.670883 10258 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:36.662477 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:36.671633 10934 raft_consensus.cc:2354] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:36.671900 10258 raft_consensus.cc:1226] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:36.671784 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.660847 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.672971 10264 pending_rounds.cc:76] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 5
I20240507 08:06:36.664621 10231 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:36.657554 10260 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:36.673228 10264 pending_rounds.cc:97] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240507 08:06:36.673555 10264 raft_consensus.cc:2866] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:36.666523 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.627869 13198 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.675892 10267 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:36.681345 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.589242 13157 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.681928 10263 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:36.682194 10304 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.680562 10190 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:36.690665 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.697134 10266 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:36.699769 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:36.700572 10332 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:36.669639 13205 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.690405 13198 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.636386 13152 consensus_queue.cc:1046] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20240507 08:06:36.690522 13282 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:36.712306 13283 raft_consensus.cc:2726] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240507 08:06:36.681731 12980 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20240507 08:06:36.710402 13304 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:36.754604 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:36.756723 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:36.779675 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 115289us
I20240507 08:06:36.767261 10263 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:36.779801 10266 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:36.780393 10231 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.781493 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:36.777278 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 95190.6us
I20240507 08:06:36.782465 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:36.784958 10304 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.787947 10304 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.790143 10333 raft_consensus.cc:1226] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:36.776424 13086 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.783411 12979 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.797993 13152 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.798672 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.798532 13152 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.788851 13168 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240507 08:06:36.786015 13292 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.785362 13283 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20240507 08:06:36.797011 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0c898839b7c84ce6ba0e135f778ad72d, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.806753 13152 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:36.809090 13290 raft_consensus.cc:2741] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20240507 08:06:36.809418 13086 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:36.791903 13282 raft_consensus.cc:2781] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:36.813760 13305 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:36.814237 10908 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:36.813799 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.813433 10258 raft_consensus.cc:1226] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20240507 08:06:36.824371 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:36.804724 13168 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:36.817530 10233 leader_election.cc:398] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849) with higher term. Message: Invalid argument: T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:36.817430 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:36.823077 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:36.824416 10265 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:36.817324 13092 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.796977 13253 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:36.834686 10333 raft_consensus.cc:1226] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:36.816531 10155 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:36.791473 13286 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20240507 08:06:36.815529 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0c898839b7c84ce6ba0e135f778ad72d, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.829082 10942 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:36.828502 10233 leader_election.cc:401] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:36.826601 13270 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:36.838656 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:36.839404 10265 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 8: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:36.839596 10304 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.840539 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.841372 10265 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240507 08:06:36.828819 13132 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=2 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240507 08:06:36.817271 13141 consensus_queue.cc:1035] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20240507 08:06:36.804013 12979 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.830269 13168 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:36.847502 10190 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20240507 08:06:36.838385 10231 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:36.851485 10267 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240507 08:06:36.851472 10155 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:36.848068 10190 pending_rounds.cc:76] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 4
I20240507 08:06:36.853847 10190 pending_rounds.cc:97] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20240507 08:06:36.854225 10190 raft_consensus.cc:2866] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:36.851761 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:36.855834 10265 raft_consensus.cc:1261] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:36.855644 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:36.858189 10188 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:36.834575 13282 raft_consensus.cc:686] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.837779 13063 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:36.817756 13292 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.844436 13274 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:36.845381 13147 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:36.847571 13270 raft_consensus.cc:686] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 8 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:36.877885 10339 raft_consensus.cc:1226] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:36.853435 13253 raft_consensus.cc:686] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:36.847838 13306 raft_consensus.cc:3037] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.837555 13092 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.864998 13063 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:36.880059 13049 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20240507 08:06:36.868943 13245 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20240507 08:06:36.888058 13132 tablet_bootstrap.cc:492] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Bootstrap complete.
I20240507 08:06:36.888962 13086 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:36.849496 13205 consensus_queue.cc:1046] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20240507 08:06:36.884521 10907 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:36.928912 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 44531.5us
I20240507 08:06:36.895817 10333 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
I20240507 08:06:36.892510 13111 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.863937 13142 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:36.866125 12980 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:36.905086 13274 raft_consensus.cc:686] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:36.917984 13308 log.cc:924] Time spent T 1f4b60e5e6294639a468641f215329a5 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20240507 08:06:36.918408 13053 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240507 08:06:36.922775 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 63ae3c4ecd5e46ec9b5cf741db56c101, TS bcf08c8276ba4a9ba646ad0f7090fcd8 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:36.941673 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 63ae3c4ecd5e46ec9b5cf741db56c101, TS bcf08c8276ba4a9ba646ad0f7090fcd8 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.881335 13147 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.868548 13152 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20240507 08:06:36.930330 13270 consensus_queue.cc:237] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:36.931185 13132 ts_tablet_manager.cc:1367] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Time spent bootstrapping tablet: real 1.316s user 0.038s sys 0.028s
I20240507 08:06:36.944718 10261 raft_consensus.cc:2354] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for earlier term 6. Current term is 7.
I20240507 08:06:36.915769 13063 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:36.904806 13042 consensus_queue.cc:1035] T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20240507 08:06:36.872092 13283 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240507 08:06:36.863812 13310 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20240507 08:06:36.818694 13275 consensus_queue.cc:1046] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20240507 08:06:36.890592 12979 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:36.921243 13286 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:36.935170 12980 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:36.905700 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:36.933534 13077 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.028s
I20240507 08:06:36.936082 13254 raft_consensus.cc:2781] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:36.945147 13147 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
W20240507 08:06:36.969688 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 103909us
I20240507 08:06:36.970957 10188 raft_consensus.cc:1226] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:36.973104 10191 raft_consensus.cc:1261] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20240507 08:06:36.943547 13142 raft_consensus.cc:686] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:36.989146 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 109502us
I20240507 08:06:36.967378 12980 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:36.940510 13282 consensus_queue.cc:237] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.931962 13198 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:36.968400 13286 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:37.028939 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 148579us
I20240507 08:06:36.991529 13254 raft_consensus.cc:686] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:37.044219 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:36.966933 13300 consensus_queue.cc:1035] T 3df41f82bc854e7c9a22f3f345b03f92 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20240507 08:06:37.046521 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:37.007221 13316 raft_consensus.cc:3032] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:37.050693 10191 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:36.960528 13253 consensus_queue.cc:237] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.074343 10267 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240507 08:06:37.004671 12979 consensus_queue.cc:1046] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20240507 08:06:37.079542 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.079860 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.082731 10190 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:37.082866 10184 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:37.084682 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.092914 10265 raft_consensus.cc:1261] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:37.026219 13042 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20240507 08:06:37.065754 13316 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:37.085378 13267 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.834s user 0.000s sys 0.006s
I20240507 08:06:37.076122 13292 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:37.105105 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 100256us
I20240507 08:06:37.113296 13319 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:37.113852 13319 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.116425 13319 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.118774 13318 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240507 08:06:37.119177 13049 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240507 08:06:37.120523 13312 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20240507 08:06:37.121212 13053 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Stepping down as leader of term 9
I20240507 08:06:37.121636 13053 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:37.122216 10265 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20240507 08:06:37.122423 13053 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.123504 13053 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:37.125880 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:37.126428 10934 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:37.130836 13152 raft_consensus.cc:3032] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Stepping down as leader of term 4
I20240507 08:06:37.131220 13152 raft_consensus.cc:729] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:37.131385 13320 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.131891 13320 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.084862 13092 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.097795 13111 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:36.994469 13274 consensus_queue.cc:237] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.131230 13198 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.130852 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1677) took 1056 ms. Trace:
I20240507 08:06:37.138680 10336 rpcz_store.cc:276] 0507 08:06:36.074357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:36.076094 (+ 1737us) service_pool.cc:225] Handling call
0507 08:06:37.130671 (+1054577us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b540006a254. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:37.130785 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":568777472}
I20240507 08:06:37.141705 13324 consensus_queue.cc:1035] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:37.142580 13132 raft_consensus.cc:348] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540 [term 8 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.132045 13152 consensus_queue.cc:260] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.132227 13049 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20240507 08:06:37.174422 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 357936us
I20240507 08:06:37.186443 10264 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20240507 08:06:37.196630 10158 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:37.203473 10156 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:37.233706 10934 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:37.235447 10233 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:37.188429 13275 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.206235 13049 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240507 08:06:37.241441 13317 log.cc:924] Time spent T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:06:37.186209 13318 consensus_queue.cc:1046] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20240507 08:06:37.205134 13115 consensus_queue.cc:1035] T 9800706282e543be9f2ee3473da2b89f P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20240507 08:06:37.101029 13142 consensus_queue.cc:237] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.183405 13324 consensus_queue.cc:1035] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20240507 08:06:37.272964 10336 raft_consensus.cc:1261] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240507 08:06:37.183808 13328 consensus_queue.cc:1046] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20240507 08:06:37.250403 13141 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:37.247357 13329 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:37.251439 13049 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.283149 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:37.285813 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.293985 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.295068 10341 raft_consensus.cc:2396] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240507 08:06:37.290599 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.296253 10340 raft_consensus.cc:2396] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20240507 08:06:37.297160 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.297981 10338 raft_consensus.cc:2362] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6. Re-sending same reply.
I20240507 08:06:37.299135 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:37.272152 13318 consensus_queue.cc:1225] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer 88a26dac37fa4243b8c2b3d47f403488 log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240507 08:06:37.298403 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.310483 10335 raft_consensus.cc:2362] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10. Re-sending same reply.
I20240507 08:06:37.302525 10190 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:37.203564 13254 consensus_queue.cc:237] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:37.283506 13132 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540 [term 8 LEARNER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Initialized, Role: LEARNER
I20240507 08:06:37.284060 13320 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.251801 13275 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.327025 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:37.300179 13049 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:37.330338 10338 raft_consensus.cc:2354] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
W20240507 08:06:37.332692 13314 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20240507 08:06:37.292752 13286 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.342161 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.352306 10340 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 5: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:37.343480 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:37.280167 13310 consensus_queue.cc:1046] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20240507 08:06:37.301349 13141 raft_consensus.cc:686] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:37.353705 10341 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 9.
I20240507 08:06:37.363062 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:37.365125 10231 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:37.367045 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:37.346104 13199 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.378306 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:37.378891 10934 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:37.379832 10906 leader_election.cc:398] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for earlier term 6. Current term is 7.
I20240507 08:06:37.368574 13198 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:37.387805 10906 leader_election.cc:401] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240507 08:06:37.388022 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:37.388727 10264 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:37.382656 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:37.390141 10258 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:37.389500 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:37.367563 13062 consensus_meta.cc:287] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.722s user 0.005s sys 0.013s
I20240507 08:06:37.372764 13120 consensus_queue.cc:1035] T e9adf1535c974b2abf910f3c1c537627 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20240507 08:06:37.367283 13245 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:37.402482 13141 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:37.403200 10304 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:37.404244 13332 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.461750 13077 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:37.461901 13332 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.409343 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:37.411916 10306 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:37.412329 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:37.464183 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 52382.1us
I20240507 08:06:37.464437 10187 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:37.464527 13332 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.464903 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:37.465163 13332 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:37.377897 13019 consensus_queue.cc:1035] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20240507 08:06:37.465751 10306 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:37.346168 13323 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20240507 08:06:37.418511 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.360677 13286 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20240507 08:06:37.384347 13125 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:37.459546 13061 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20240507 08:06:37.459942 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 56854.6us
I20240507 08:06:37.460026 13063 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.462322 13077 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:37.420792 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 63520.8us
I20240507 08:06:37.458684 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.426455 10191 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
W20240507 08:06:37.425843 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 46239.9us
I20240507 08:06:37.486284 10231 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:37.465519 13332 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.436180 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:37.426863 13168 consensus_meta.cc:287] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.561s user 0.009s sys 0.008s
I20240507 08:06:37.489620 10259 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:37.447831 10258 raft_consensus.cc:2454] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:37.481865 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:37.382452 13199 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.413503 13275 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.435508 10190 raft_consensus.cc:1261] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20240507 08:06:37.470598 13092 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:37.494342 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:37.497318 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 248964us
I20240507 08:06:37.514979 10266 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:37.516494 10231 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:37.517418 10189 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:37.402536 13198 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.367506 13132 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } attrs { replace: true } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: NON_VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: true } }
I20240507 08:06:37.482368 13061 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.434916 13344 raft_consensus.cc:2726] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5d6656a616b747d393ff3de5328d2540 for earlier term 6. Current term is 7.
W20240507 08:06:37.531314 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 43860.1us
I20240507 08:06:37.472837 13337 raft_consensus.cc:2726] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:37.514452 13121 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:37.491557 13332 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:37.467159 13245 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:37.436257 13062 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.515724 13092 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.554571 10262 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:37.562242 13199 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.564275 13157 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:37.564854 13306 consensus_meta.cc:287] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.639s user 0.003s sys 0.011s
I20240507 08:06:37.565261 13306 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849) with higher term. Message: Invalid argument: T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:37.535591 13061 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:37.533099 13169 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:37.553994 13121 raft_consensus.cc:686] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:37.576176 10306 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:37.554123 13092 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:37.515700 13168 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.584019 10263 raft_consensus.cc:1226] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 2. Current term is 3. Ops: []
I20240507 08:06:37.585227 10306 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:37.589183 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 72792.1us
W20240507 08:06:37.590735 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 91898.3us
I20240507 08:06:37.596634 10231 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:37.599001 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.599856 10191 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:37.583994 13359 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.609602 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.610251 10183 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:37.609961 10306 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:37.611305 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:37.587874 13253 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:37.620007 10231 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:37.625756 13343 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.076s user 0.003s sys 0.000s
I20240507 08:06:37.643095 10183 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:37.608711 13286 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240507 08:06:37.609269 13169 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:37.682174 13269 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:37.682708 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 82067.4us
I20240507 08:06:37.685853 13328 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:37.686355 13328 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.686775 13328 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:37.620220 13337 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240507 08:06:37.689806 13337 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.690259 13337 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:37.611512 13359 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.684676 13086 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:37.720150 13086 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:37.695628 13132 ts_tablet_manager.cc:1398] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Time spent starting tablet: real 0.750s user 0.011s sys 0.000s
I20240507 08:06:37.720564 13086 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:37.720721 13132 ts_tablet_manager.cc:1430] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Tablet startup took 775ms
I20240507 08:06:37.722867 13117 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:37.721132 13132 ts_tablet_manager.cc:1432] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Trace:
0507 08:06:32.474961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:32.475335 (+ 374us) service_pool.cc:225] Handling call
0507 08:06:32.892162 (+416827us) ts_tablet_manager.cc:903] T 0c898839b7c84ce6ba0e135f778ad72d P 5d6656a616b747d393ff3de5328d2540: Initiating tablet copy from peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
0507 08:06:33.846245 (+954083us) tablet_copy_client.cc:361] Tablet copy session begun
0507 08:06:34.388841 (+542596us) tablet_metadata.cc:641] Metadata flushed
0507 08:06:34.454559 (+ 65718us) tablet_copy_client.cc:490] Wrote new tablet metadata
0507 08:06:34.742128 (+287569us) tablet_copy_client.cc:411] Wrote new consensus metadata
0507 08:06:34.742568 (+ 440us) tablet_replica.cc:205] Creating consensus instance
0507 08:06:34.742940 (+ 372us) ts_tablet_manager.cc:958] Replica registered.
0507 08:06:34.743050 (+ 110us) inbound_call.cc:173] Queueing success response
0507 08:06:35.585953 (+842903us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0507 08:06:36.945865 (+1359912us) ts_tablet_manager.cc:1399] Starting tablet replica
0507 08:06:36.961494 (+ 15629us) tablet_replica.cc:259] Starting instrumentation
0507 08:06:37.077068 (+115574us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":237,"raft.queue_time_us":5403,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":11389,"tablet-copy.queue_time_us":392125,"tablet-download-pool.queue_time_us":1498,"tablet-download-pool.run_cpu_time_us":1112,"tablet-download-pool.run_wall_time_us":162467,"thread_start_us":374727,"threads_started":2}
I20240507 08:06:37.723374 13117 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:37.725054 12980 consensus_meta.cc:287] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.716s user 0.005s sys 0.005s
W20240507 08:06:37.726128 13357 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20240507 08:06:37.696614 13063 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.607581 13358 consensus_queue.cc:1035] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
W20240507 08:06:37.728044 13338 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.317s user 0.000s sys 0.001s
I20240507 08:06:37.727763 13049 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:37.728200 13019 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:37.728039 13063 consensus_queue.cc:1046] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:37.723258 13318 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.725468 12980 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.730028 13318 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.730967 13359 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.731572 12980 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.733218 13049 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:37.735577 13289 raft_consensus.cc:3032] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:37.735947 13289 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:37.707069 13062 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.694053 13121 consensus_queue.cc:237] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.716414 13198 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:37.716794 13147 consensus_meta.cc:287] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.725s user 0.008s sys 0.003s
W20240507 08:06:37.718194 13349 log.cc:924] Time spent T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.143s user 0.000s sys 0.002s
I20240507 08:06:37.718497 13326 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:37.736464 13061 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.859875 13205 raft_consensus.cc:3032] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:37.753230 13269 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.871362 13205 raft_consensus.cc:729] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:37.871800 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 262794us
I20240507 08:06:37.804821 13328 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.816403 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2182) took 1488 ms. Trace:
I20240507 08:06:37.875031 13328 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.873924 10192 rpcz_store.cc:276] 0507 08:06:36.327850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:36.329523 (+ 1673us) service_pool.cc:225] Handling call
0507 08:06:36.333608 (+ 4085us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:36.350670 (+ 17062us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:36.350698 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.141398 (+790700us) spinlock_profiling.cc:243] Waited 929 us on lock 0x7b1800353f80. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8decab06 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8f809f11 00007fcf8f819542 00007fcf90e62fc1 00007fcf90e277da
0507 08:06:37.142668 (+ 1270us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b2c00000918. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8f809f11 00007fcf8f819542 00007fcf90e62fc1 00007fcf90e277da 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782
0507 08:06:37.143092 (+ 424us) log.cc:842] Serialized 26 byte log entry
0507 08:06:37.524885 (+381793us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:37.524927 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:37.524984 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:37.816195 (+291211us) raft_consensus.cc:1671] finished
0507 08:06:37.816217 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:37.816330 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4545,"raft.queue_time_us":12784,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":12869,"spinlock_wait_cycles":5764224,"thread_start_us":347516,"threads_started":1,"wal-append.queue_time_us":671363}
W20240507 08:06:37.753674 13053 consensus_meta.cc:287] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.630s user 0.012s sys 0.000s
I20240507 08:06:37.753468 13310 consensus_queue.cc:1046] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.123s
I20240507 08:06:37.844312 13316 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.791213 10089 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47176 (request call id 123) took 2887 ms. Trace:
W20240507 08:06:37.822849 13341 log.cc:924] Time spent T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.409s user 0.000s sys 0.001s
I20240507 08:06:37.817502 13337 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.877635 10089 rpcz_store.cc:276] 0507 08:06:34.903721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:34.904011 (+ 290us) service_pool.cc:225] Handling call
0507 08:06:37.791121 (+2887110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:35.557046 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:35.563145 (+ 6099us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:35.563197 (+ 52us) write_op.cc:435] Acquired schema lock
0507 08:06:35.563211 (+ 14us) tablet.cc:660] Decoding operations
0507 08:06:35.595113 (+ 31902us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:35.595191 (+ 78us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:35.595217 (+ 26us) tablet.cc:683] Acquiring locks for 7 operations
0507 08:06:35.596018 (+ 801us) tablet.cc:699] Row locks acquired
0507 08:06:35.596035 (+ 17us) write_op.cc:260] PREPARE: finished
0507 08:06:35.596214 (+ 179us) write_op.cc:270] Start()
0507 08:06:35.596365 (+ 151us) write_op.cc:276] Timestamp: P: 1715069195596149 usec, L: 0
0507 08:06:35.596382 (+ 17us) op_driver.cc:331] REPLICATION: starting
0507 08:06:35.597011 (+ 629us) log.cc:842] Serialized 2835 byte log entry
0507 08:06:36.004898 (+407887us) op_driver.cc:444] REPLICATION: finished
0507 08:06:36.107246 (+102348us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf8dedf16e 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf94838dd0
0507 08:06:36.284275 (+177029us) write_op.cc:301] APPLY: starting
0507 08:06:36.287270 (+ 2995us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:36.342672 (+ 55402us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:36.342718 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:36.459375 (+116657us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:36.459603 (+ 228us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0507 08:06:36.459666 (+ 63us) write_op.cc:312] APPLY: finished
0507 08:06:36.459924 (+ 258us) log.cc:842] Serialized 59 byte log entry
0507 08:06:36.460068 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:36.460175 (+ 107us) write_op.cc:454] Released schema lock
0507 08:06:36.460225 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:36.460486 (+ 261us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:36.460512 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":52593,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":271638,"apply.run_cpu_time_us":3528,"apply.run_wall_time_us":180816,"num_ops":7,"prepare.queue_time_us":343187,"prepare.run_cpu_time_us":3918,"prepare.run_wall_time_us":40250,"replication_time_us":408415,"spinlock_wait_cycles":108423552,"thread_start_us":434672,"threads_started":2}]]}
I20240507 08:06:37.880802 13332 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
W20240507 08:06:37.877954 13356 log.cc:924] Time spent T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.270s user 0.002s sys 0.000s
I20240507 08:06:37.881883 12867 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:37.708118 13276 raft_consensus.cc:3032] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:37.872530 13205 consensus_queue.cc:260] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.882644 13147 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.883694 13152 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:37.883728 13300 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:37.877139 13316 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:37.884634 13329 raft_consensus.cc:3032] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:37.884833 13326 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:37.887142 13086 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:37.811735 13092 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:37.889498 13061 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.877139 13310 raft_consensus.cc:3032] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Stepping down as leader of term 2
I20240507 08:06:37.866852 13117 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.891628 13269 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:37.893311 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:37.872578 10231 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:37.896445 13333 consensus_queue.cc:1046] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:37.896973 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:38.074967 10304 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.003s sys 0.018s
W20240507 08:06:38.075382 10304 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.543s user 0.003s sys 0.018s
I20240507 08:06:38.075309 10183 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:37.898315 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:37.902108 10185 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:37.896889 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:37.880371 13337 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:37.911334 13335 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.498s user 0.002s sys 0.000s
I20240507 08:06:37.891794 13318 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.926335 13077 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:37.928515 13289 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:37.937682 13168 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:38.052727 12979 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:38.065552 10158 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.004s sys 0.014s
W20240507 08:06:38.110535 10158 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.005s sys 0.014s
I20240507 08:06:37.949492 13276 raft_consensus.cc:729] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:38.075866 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 543372us
I20240507 08:06:38.121279 10304 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:38.080745 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:38.122221 10192 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Advancing to term 11
W20240507 08:06:38.079844 13314 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20240507 08:06:38.068743 13310 raft_consensus.cc:729] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:38.076705 10190 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:38.084736 10333 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:38.086143 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:38.071471 13083 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.941s user 0.008s sys 0.017s
I20240507 08:06:37.986729 13329 raft_consensus.cc:729] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:38.072073 13205 raft_consensus.cc:3037] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:38.082029 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 209617us
I20240507 08:06:38.130259 10339 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:38.132151 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:38.136126 10185 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:38.138079 10231 leader_election.cc:398] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:38.143280 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:38.140587 10156 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:38.138998 10231 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:38.144593 10339 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:38.118053 13333 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:38.135216 13083 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.002s user 0.009s sys 0.017s
I20240507 08:06:38.155028 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:38.161455 10262 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:38.155905 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:38.157533 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:38.136870 13282 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.014s sys 0.045s
I20240507 08:06:38.247995 10259 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20240507 08:06:38.144307 13344 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.011s sys 0.030s
I20240507 08:06:38.251291 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:37.981277 13300 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:38.196592 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 56100.3us
I20240507 08:06:38.131969 13318 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:38.268078 10260 raft_consensus.cc:2379] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 6: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
W20240507 08:06:38.272078 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 64632a3e07184039950f3ef7fe699cb3, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:38.251456 13344 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.621s user 0.011s sys 0.030s
I20240507 08:06:38.315418 13300 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:38.273145 10339 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:38.316197 10336 raft_consensus.cc:1261] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20240507 08:06:38.273476 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 116118us
I20240507 08:06:38.160917 13289 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:38.247469 13282 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.975s user 0.014s sys 0.045s
I20240507 08:06:38.145993 13061 raft_consensus.cc:2781] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:38.122881 12979 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:38.282148 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 64632a3e07184039950f3ef7fe699cb3, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.282719 10258 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20240507 08:06:38.282734 10262 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:38.326331 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:38.286921 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:38.287492 13361 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:38.152987 13305 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:38.174285 13121 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:38.288429 13306 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:38.248730 13333 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.289364 13336 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:38.294942 13318 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.295841 13320 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:38.296397 13322 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:38.244192 13157 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:38.153137 10183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:38.345533 10183 raft_consensus.cc:595] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:38.298703 13092 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:38.300999 13245 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:38.302626 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:38.302803 13269 raft_consensus.cc:3032] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:38.303844 13325 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:38.304423 13315 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:38.310158 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 404) took 1297 ms. Trace:
I20240507 08:06:38.313876 13326 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.314823 13310 consensus_queue.cc:260] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.251793 13276 consensus_queue.cc:260] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.323222 10156 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:38.331802 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:38.484545 10339 raft_consensus.cc:2354] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10.
I20240507 08:06:38.332463 10332 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
W20240507 08:06:38.333000 10906 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.006s sys 0.015s
W20240507 08:06:38.485453 10906 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.006s sys 0.015s
I20240507 08:06:38.341594 13306 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.487378 13306 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:38.488881 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:38.489567 10942 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
W20240507 08:06:38.337237 10261 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.063s sys 0.090s
I20240507 08:06:38.337688 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:38.490952 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:38.345940 10183 raft_consensus.cc:603] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240507 08:06:38.339998 10155 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.822s user 0.080s sys 0.130s
I20240507 08:06:38.356400 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:38.361554 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:38.342161 13332 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:38.345337 13157 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:38.341775 13336 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:38.322283 10185 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:38.350637 13147 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:38.422070 13325 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:38.434640 13333 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:38.336150 13361 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.430299 13310 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 2 FOLLOWER]: Advancing to term 3
I20240507 08:06:38.423110 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:38.347515 13322 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:38.335800 13305 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:38.428908 13368 log.cc:924] Time spent T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20240507 08:06:38.363232 10337 rpcz_store.cc:276] 0507 08:06:37.012433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.012593 (+ 160us) service_pool.cc:225] Handling call
0507 08:06:37.012990 (+ 397us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.052112 (+ 39122us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.052157 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.065314 (+ 13157us) log.cc:842] Serialized 26 byte log entry
0507 08:06:37.261011 (+195697us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:37.261050 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:37.261107 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:37.910674 (+649567us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:38.309857 (+399183us) raft_consensus.cc:1671] finished
0507 08:06:38.309879 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:38.310075 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38612,"raft.queue_time_us":99727,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":21927680,"thread_start_us":170161,"threads_started":1,"wal-append.queue_time_us":299949}
I20240507 08:06:38.435689 12979 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:38.352699 13269 raft_consensus.cc:729] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:38.343855 13061 raft_consensus.cc:686] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:38.436390 13318 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:38.367933 13086 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:38.347635 13320 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:38.340227 13329 consensus_queue.cc:260] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:39.226027 10155 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.708s user 0.087s sys 0.141s
I20240507 08:06:38.491283 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:38.840070 10267 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.000s sys 0.026s
W20240507 08:06:39.226573 10267 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s user 0.001s sys 0.037s
W20240507 08:06:38.517092 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 194039us
I20240507 08:06:38.489817 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
W20240507 08:06:38.966615 10192 consensus_meta.cc:287] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.844s user 0.006s sys 0.009s
W20240507 08:06:38.542142 10906 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.8.213.190:37115, user_credentials={real_user=slave}} blocked reactor thread for 53056.6us
I20240507 08:06:39.228956 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:39.229668 10258 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
W20240507 08:06:39.230023 10261 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.808s user 0.072s sys 0.110s
W20240507 08:06:38.888612 10190 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.014s sys 0.014s
W20240507 08:06:39.226423 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 1.73286e+06us
W20240507 08:06:39.243019 10190 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.939s user 0.017s sys 0.026s
I20240507 08:06:39.232720 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:38.626322 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:38.853902 10332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 2118) took 2030 ms (client timeout 2240 ms). Trace:
I20240507 08:06:39.229952 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 380) took 1333 ms. Trace:
I20240507 08:06:38.944582 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:39.247000 10336 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:39.247789 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 2133) took 1299 ms. Trace:
W20240507 08:06:39.232149 10942 consensus_meta.cc:287] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.742s user 0.019s sys 0.005s
I20240507 08:06:39.248292 10336 rpcz_store.cc:276] 0507 08:06:37.948208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.841052 (+892844us) service_pool.cc:225] Handling call
0507 08:06:39.247713 (+406661us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.248512 10942 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:39.253598 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2223) took 1115 ms. Trace:
I20240507 08:06:39.244518 10192 raft_consensus.cc:2454] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 11.
I20240507 08:06:39.261785 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2222) took 1238 ms. Trace:
W20240507 08:06:39.245128 10332 rpcz_store.cc:269] 0507 08:06:36.823880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:36.824023 (+ 143us) service_pool.cc:225] Handling call
0507 08:06:38.853695 (+2029672us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b5400049f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:38.853830 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3468120320}
I20240507 08:06:39.262125 10192 rpcz_store.cc:276] 0507 08:06:38.023206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.023743 (+ 537us) service_pool.cc:225] Handling call
0507 08:06:39.261726 (+1237983us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.260744 10185 rpcz_store.cc:276] 0507 08:06:38.112708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.113010 (+ 302us) service_pool.cc:225] Handling call
0507 08:06:39.228175 (+1115165us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.266582 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:51764 (request call id 29) took 1108 ms. Trace:
I20240507 08:06:39.267880 10183 rpcz_store.cc:276] 0507 08:06:38.123635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.124634 (+ 999us) service_pool.cc:225] Handling call
0507 08:06:39.231598 (+1106964us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:38.652673 13346 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.011s sys 0.009s
W20240507 08:06:39.078006 13382 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.729s user 0.000s sys 0.001s
W20240507 08:06:39.101949 13316 consensus_meta.cc:287] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 1.120s user 0.007s sys 0.003s
W20240507 08:06:38.883898 13327 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s user 0.023s sys 0.023s
W20240507 08:06:38.649111 13385 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.219s user 0.001s sys 0.000s
I20240507 08:06:38.483495 13337 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.270648 13141 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.253448 13325 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.274310 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:39.303778 10185 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:38.417642 13315 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:38.491605 13274 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.278523 13147 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:39.278959 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 46172.5us
I20240507 08:06:39.279709 13320 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.280726 13198 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.285481 12979 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.285914 13169 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.286087 13168 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:39.173303 13279 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s user 0.008s sys 0.010s
I20240507 08:06:39.288897 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:39.382579 10192 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:39.383313 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1835) took 1108 ms. Trace:
I20240507 08:06:39.383806 10192 rpcz_store.cc:276] 0507 08:06:38.275148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.288649 (+1013501us) service_pool.cc:225] Handling call
0507 08:06:39.383240 (+ 94591us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.384930 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:39.250819 10189 rpcz_store.cc:276] 0507 08:06:37.896066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.897291 (+ 1225us) service_pool.cc:225] Handling call
0507 08:06:37.897709 (+ 418us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:06:37.898153 (+ 444us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.898189 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:06:37.898698 (+ 509us) log.cc:842] Serialized 52 byte log entry
0507 08:06:38.119144 (+220446us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:38.119228 (+ 84us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.119735 (+ 507us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:38.527524 (+407789us) raft_consensus.cc:1671] finished
0507 08:06:38.527549 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.229898 (+702349us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1997,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":219967,"threads_started":1,"wal-append.queue_time_us":519673}
I20240507 08:06:39.386427 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:39.387123 10189 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:39.388845 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1833) took 1250 ms. Trace:
I20240507 08:06:39.288223 10087 catalog_manager.cc:5484] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 7, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:39.292954 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:38.971889 13374 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.224177 13319 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.231932 13157 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:39.178838 13381 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.315s user 0.000s sys 0.001s
I20240507 08:06:39.230567 13276 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:38.953797 13282 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:38.872079 13280 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:39.318790 13141 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:39.398188 13372 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20240507 08:06:39.260320 13336 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:38.543628 13367 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:38.733233 13290 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.297050 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:39.297042 10191 raft_consensus.cc:1261] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20240507 08:06:39.427650 10191 pending_rounds.cc:76] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 4
I20240507 08:06:39.427951 10191 pending_rounds.cc:97] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240507 08:06:39.428254 10191 raft_consensus.cc:2866] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:39.138298 13375 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:39.301373 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:39.302141 10934 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:39.277848 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:39.317778 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:39.293649 13327 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.148s user 0.031s sys 0.027s
I20240507 08:06:39.268735 13318 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:39.323392 10185 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
W20240507 08:06:39.268728 13115 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.990s user 0.031s sys 0.137s
I20240507 08:06:39.324352 10258 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
W20240507 08:06:39.300695 13344 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.985s user 0.014s sys 0.039s
I20240507 08:06:39.332142 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1956) took 1177 ms. Trace:
I20240507 08:06:39.304160 13316 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.346150 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1954) took 1195 ms. Trace:
I20240507 08:06:39.442670 10263 rpcz_store.cc:276] 0507 08:06:38.151061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.151212 (+ 151us) service_pool.cc:225] Handling call
0507 08:06:38.267649 (+116437us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.289617 (+1021968us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.289659 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.289716 (+ 57us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:39.289760 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.328327 (+ 38567us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002bfa54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:39.328387 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.328549 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2350926208}
I20240507 08:06:39.339824 13198 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.343647 10304 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:39.446971 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 103497us
I20240507 08:06:39.324154 13329 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:39.390595 10336 raft_consensus.cc:2379] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 8: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:39.392930 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 381) took 1496 ms. Trace:
I20240507 08:06:39.391707 10087 catalog_manager.cc:5484] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 4 to 5, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 5 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:39.454910 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:39.331389 13269 consensus_queue.cc:260] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.392067 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2125) took 1499 ms. Trace:
I20240507 08:06:39.401965 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2225) took 1240 ms. Trace:
W20240507 08:06:39.390839 13279 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.104s user 0.012s sys 0.010s
I20240507 08:06:39.304405 13337 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:39.391520 13380 log.cc:924] Time spent T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.546s user 0.001s sys 0.000s
W20240507 08:06:39.391381 13391 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.504s user 0.000s sys 0.001s
I20240507 08:06:38.516813 13306 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.421849 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 493) took 1885 ms. Trace:
I20240507 08:06:39.500396 10266 rpcz_store.cc:276] 0507 08:06:37.536908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.537100 (+ 192us) service_pool.cc:225] Handling call
0507 08:06:37.537491 (+ 391us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.544323 (+ 6832us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.544360 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.566691 (+ 22331us) log.cc:842] Serialized 26 byte log entry
0507 08:06:37.998177 (+431486us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:38.074113 (+ 75936us) log.cc:842] Serialized 14 byte log entry
0507 08:06:38.074337 (+ 224us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.074425 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:38.310659 (+236234us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b1800448660. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd1b8 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:06:39.414607 (+1103948us) raft_consensus.cc:1671] finished
0507 08:06:39.421477 (+ 6870us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.421684 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":83064,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":11919,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":14379,"spinlock_wait_cycles":5629696,"thread_start_us":409291,"threads_started":1,"wal-append.queue_time_us":560720}
I20240507 08:06:39.398842 13399 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:39.398341 13377 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.429607 10183 raft_consensus.cc:2454] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 5.
I20240507 08:06:39.513558 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50108 (request call id 382) took 1219 ms. Trace:
I20240507 08:06:39.478049 13364 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:39.439116 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2229) took 1147 ms. Trace:
W20240507 08:06:39.443658 10265 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.082s user 0.015s sys 0.085s
W20240507 08:06:39.514897 10265 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.158s user 0.015s sys 0.085s
I20240507 08:06:39.514788 10185 rpcz_store.cc:276] 0507 08:06:38.292172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.274061 (+981889us) service_pool.cc:225] Handling call
0507 08:06:39.439058 (+164997us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.334602 13147 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.449018 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:39.381001 12979 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.445091 10267 rpcz_store.cc:276] 0507 08:06:38.150371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.151722 (+ 1351us) service_pool.cc:225] Handling call
0507 08:06:38.152106 (+ 384us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:38.269694 (+117588us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:38.269740 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:38.270273 (+ 533us) log.cc:842] Serialized 26 byte log entry
0507 08:06:39.227103 (+956830us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:39.227140 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.227205 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.345825 (+118620us) raft_consensus.cc:1671] finished
0507 08:06:39.345855 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.346057 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5123,"raft.queue_time_us":114991,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":15117,"thread_start_us":929102,"threads_started":1,"wal-append.queue_time_us":1046385}
I20240507 08:06:39.453461 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1965) took 1095 ms. Trace:
I20240507 08:06:39.455545 10087 catalog_manager.cc:5484] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 5 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:39.389235 10189 rpcz_store.cc:276] 0507 08:06:38.138214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.387946 (+1249732us) service_pool.cc:225] Handling call
0507 08:06:39.388284 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.388490 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.388526 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.388582 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:39.388624 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.388691 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.388800 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.390650 13169 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.392468 13168 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.455513 10260 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:39.531977 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 1992) took 2174 ms. Trace:
I20240507 08:06:39.549846 10263 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
W20240507 08:06:39.463811 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 281283us
I20240507 08:06:39.482914 10192 rpcz_store.cc:276] 0507 08:06:38.161133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.400687 (+1239554us) service_pool.cc:225] Handling call
0507 08:06:39.401028 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.401231 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.401267 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.401321 (+ 54us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:39.401364 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.401430 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.401552 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.440632 13290 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.485997 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39714 (request call id 2131) took 1538 ms. Trace:
I20240507 08:06:39.488031 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2124) took 1595 ms. Trace:
I20240507 08:06:39.559198 10336 rpcz_store.cc:276] 0507 08:06:37.947669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.292695 (+1345026us) service_pool.cc:225] Handling call
0507 08:06:39.485925 (+193230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.400310 13374 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.483243 10338 rpcz_store.cc:276] 0507 08:06:37.892656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.894473 (+ 1817us) service_pool.cc:225] Handling call
0507 08:06:37.894865 (+ 392us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.895273 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.895311 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.895793 (+ 482us) log.cc:842] Serialized 26 byte log entry
0507 08:06:38.083100 (+187307us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:38.083253 (+ 153us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.083542 (+ 289us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.391842 (+1308300us) raft_consensus.cc:1671] finished
0507 08:06:39.391866 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.392009 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":116,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":185355,"threads_started":1,"wal-append.queue_time_us":448801}
W20240507 08:06:39.444615 13115 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.172s user 0.031s sys 0.143s
W20240507 08:06:39.524897 13205 consensus_meta.cc:287] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 1.381s user 0.012s sys 0.000s
W20240507 08:06:39.442615 13344 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.127s user 0.014s sys 0.039s
I20240507 08:06:39.564477 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2123) took 1672 ms. Trace:
W20240507 08:06:39.494175 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 215056us
I20240507 08:06:39.403059 13319 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.424963 13280 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.514034 10183 rpcz_store.cc:276] 0507 08:06:38.293779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.276831 (+983052us) service_pool.cc:225] Handling call
0507 08:06:39.513485 (+236654us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.427160 10334 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:39.569319 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1836) took 1285 ms. Trace:
I20240507 08:06:39.569625 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1697) took 1081 ms. Trace:
I20240507 08:06:39.490847 13332 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.400415 13061 consensus_queue.cc:237] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.420290 13282 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.532478 10087 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 6 to 7, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 7 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:39.531608 10258 rpcz_store.cc:276] 0507 08:06:38.347909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.348044 (+ 135us) service_pool.cc:225] Handling call
0507 08:06:39.439246 (+1091202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.446888 13318 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.454439 10184 rpcz_store.cc:276] 0507 08:06:37.896330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.899353 (+ 3023us) service_pool.cc:225] Handling call
0507 08:06:37.899634 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.899978 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.900013 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.900377 (+ 364us) log.cc:842] Serialized 26 byte log entry
0507 08:06:38.120234 (+219857us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:38.120314 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.120414 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.392749 (+1272335us) raft_consensus.cc:1671] finished
0507 08:06:39.392770 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.392878 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1351,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":219381,"threads_started":1,"wal-append.queue_time_us":442508}
I20240507 08:06:39.554316 10306 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:39.499356 13269 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:39.501854 13305 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.455936 13316 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.330116 13274 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.569885 10192 rpcz_store.cc:276] 0507 08:06:38.283693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.553395 (+1269702us) service_pool.cc:225] Handling call
0507 08:06:39.568664 (+ 15269us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.568909 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.568955 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.569013 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:39.569057 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.569127 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.569252 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.550282 10261 rpcz_store.cc:276] 0507 08:06:37.340596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.340740 (+ 144us) service_pool.cc:225] Handling call
0507 08:06:37.356833 (+ 16093us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.370342 (+ 13509us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.370384 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.378210 (+ 7826us) log.cc:842] Serialized 26 byte log entry
0507 08:06:39.245762 (+1867552us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:39.245817 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.245905 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.514711 (+268806us) raft_consensus.cc:1671] finished
0507 08:06:39.514738 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.514873 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13110,"raft.queue_time_us":15944,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":3499,"thread_start_us":1823308,"threads_started":1,"wal-append.queue_time_us":1313970}
I20240507 08:06:39.523726 13364 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:39.495007 13333 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.064s
I20240507 08:06:39.576045 10188 raft_consensus.cc:1261] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20240507 08:06:39.597182 10188 pending_rounds.cc:76] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 3
I20240507 08:06:39.597512 10188 pending_rounds.cc:97] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20240507 08:06:39.597842 10188 raft_consensus.cc:2866] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:39.570098 10334 rpcz_store.cc:276] 0507 08:06:38.488181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.271127 (+782946us) service_pool.cc:225] Handling call
0507 08:06:39.569541 (+298414us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.559391 10341 rpcz_store.cc:276] 0507 08:06:37.892385 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.892497 (+ 112us) service_pool.cc:225] Handling call
0507 08:06:37.892876 (+ 379us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.893247 (+ 371us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.893274 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.893760 (+ 486us) log.cc:842] Serialized 26 byte log entry
0507 08:06:38.244494 (+350734us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:38.244553 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.244640 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.487743 (+1243103us) raft_consensus.cc:1671] finished
0507 08:06:39.487775 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.487929 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1973,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"thread_start_us":265018,"threads_started":1,"wal-append.queue_time_us":685247}
I20240507 08:06:39.582573 10259 raft_consensus.cc:2396] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20240507 08:06:39.566100 10335 rpcz_store.cc:276] 0507 08:06:37.892062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.892198 (+ 136us) service_pool.cc:225] Handling call
0507 08:06:37.892488 (+ 290us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.892804 (+ 316us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.892855 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.893277 (+ 422us) log.cc:842] Serialized 26 byte log entry
0507 08:06:38.118352 (+225075us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:38.118473 (+ 121us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.118640 (+ 167us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.564220 (+1445580us) raft_consensus.cc:1671] finished
0507 08:06:39.564250 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.564409 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2178,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":224489,"threads_started":1,"wal-append.queue_time_us":469510}
I20240507 08:06:39.581609 10087 catalog_manager.cc:5484] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:39.584570 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 414) took 1637 ms. Trace:
I20240507 08:06:39.550137 13322 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.440829 13375 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.494799 13401 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20240507 08:06:39.591259 10263 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:39.592944 10333 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:39.512809 13306 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.591120 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 415) took 1643 ms. Trace:
I20240507 08:06:39.618446 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 417) took 1477 ms. Trace:
I20240507 08:06:39.623881 10338 rpcz_store.cc:276] 0507 08:06:37.947359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.568147 (+1620788us) service_pool.cc:225] Handling call
0507 08:06:39.590493 (+ 22346us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.590720 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.590757 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.590812 (+ 55us) raft_consensus.cc:1631] Marking committed up to 10
0507 08:06:39.590855 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.590933 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.591055 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.624194 10335 rpcz_store.cc:276] 0507 08:06:38.140705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.601339 (+1460634us) service_pool.cc:225] Handling call
0507 08:06:39.617881 (+ 16542us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.618074 (+ 193us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.618109 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.618158 (+ 49us) raft_consensus.cc:1631] Marking committed up to 9
0507 08:06:39.618199 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.618264 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.618384 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.554155 13377 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:39.596442 13152 consensus_meta.cc:287] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 1.616s user 0.007s sys 0.006s
I20240507 08:06:39.600199 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2136) took 1460 ms. Trace:
I20240507 08:06:39.593982 13205 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.560703 13092 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20240507 08:06:39.618634 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2132) took 1670 ms. Trace:
I20240507 08:06:39.595803 13325 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.598246 13305 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.616815 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 383) took 1322 ms. Trace:
I20240507 08:06:39.628334 10339 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:39.633677 10341 rpcz_store.cc:276] 0507 08:06:38.140196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.599209 (+1459013us) service_pool.cc:225] Handling call
0507 08:06:39.599567 (+ 358us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.599780 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.599820 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.599874 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:39.599930 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.600000 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.600131 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.639184 10192 rpcz_store.cc:276] 0507 08:06:38.294042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.595285 (+1301243us) service_pool.cc:225] Handling call
0507 08:06:39.613144 (+ 17859us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.613374 (+ 230us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.613408 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.613461 (+ 53us) raft_consensus.cc:1631] Marking committed up to 10
0507 08:06:39.613507 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.616592 (+ 3085us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.616748 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.611042 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2228) took 1319 ms. Trace:
I20240507 08:06:39.645721 13141 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.646512 13147 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.646857 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2219) took 1747 ms. Trace:
I20240507 08:06:39.647104 13147 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:39.647943 13169 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:39.636271 13378 log.cc:924] Time spent T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.303s user 0.001s sys 0.000s
I20240507 08:06:39.531693 13399 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.635627 13315 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.674521 13374 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.597910 13361 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.638154 10334 rpcz_store.cc:276] 0507 08:06:37.947932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.598979 (+1651047us) service_pool.cc:225] Handling call
0507 08:06:39.618026 (+ 19047us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.618250 (+ 224us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.618286 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.618343 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:39.618385 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.618449 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.618570 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.676285 10334 raft_consensus.cc:1261] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240507 08:06:39.676610 10334 pending_rounds.cc:76] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Aborting all ops after (but not including) 3
I20240507 08:06:39.676816 10334 pending_rounds.cc:97] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240507 08:06:39.677050 10334 raft_consensus.cc:2866] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:39.677652 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1694) took 1352 ms. Trace:
I20240507 08:06:39.677942 10334 rpcz_store.cc:276] 0507 08:06:38.325102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.675633 (+1350531us) service_pool.cc:225] Handling call
0507 08:06:39.675943 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.677419 (+ 1476us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.677601 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.640504 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1692) took 1319 ms. Trace:
I20240507 08:06:39.680503 10341 rpcz_store.cc:276] 0507 08:06:38.320860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.639587 (+1318727us) service_pool.cc:225] Handling call
0507 08:06:39.639923 (+ 336us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.640124 (+ 201us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.640163 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.640214 (+ 51us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:39.640256 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.640321 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.640444 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.646781 13141 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.682006 13141 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.683873 13141 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.684286 13141 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.684726 13141 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.621871 13322 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:39.634932 13395 log.cc:924] Time spent T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.236s user 0.000s sys 0.001s
I20240507 08:06:39.687610 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 413) took 1740 ms. Trace:
I20240507 08:06:39.687907 10337 rpcz_store.cc:276] 0507 08:06:37.946762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.443404 (+1496642us) service_pool.cc:225] Handling call
0507 08:06:39.479401 (+ 35997us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.577648 (+ 98247us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.577690 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.577745 (+ 55us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:39.577790 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.687456 (+109666us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b54003c3f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:39.687488 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.687570 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225525632}
I20240507 08:06:39.647351 10188 rpcz_store.cc:276] 0507 08:06:37.899838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.901500 (+ 1662us) service_pool.cc:225] Handling call
0507 08:06:37.901820 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.598528 (+1696708us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.646653 (+ 48125us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54001bf3d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:39.646805 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3851272960}
I20240507 08:06:39.648486 13169 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.574856 13327 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:39.649787 10338 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:39.698249 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 418) took 1410 ms. Trace:
I20240507 08:06:39.698603 10338 rpcz_store.cc:276] 0507 08:06:38.288071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.625656 (+1337585us) service_pool.cc:225] Handling call
0507 08:06:39.637315 (+ 11659us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.697926 (+ 60611us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.698181 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.604652 10336 rpcz_store.cc:276] 0507 08:06:37.947069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.566561 (+1619492us) service_pool.cc:225] Handling call
0507 08:06:39.583976 (+ 17415us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.584193 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.584230 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.584285 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:39.584329 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.584396 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.584519 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.597365 13364 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:39.603489 10087 catalog_manager.cc:5484] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 8 to 9, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 9 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:39.640965 10184 rpcz_store.cc:276] 0507 08:06:38.291886 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.583121 (+1291235us) service_pool.cc:225] Handling call
0507 08:06:39.610444 (+ 27323us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.610642 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.610678 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.610733 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:39.610776 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.610844 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.610976 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.674672 13306 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.587939 13332 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.647624 13147 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.636648 10335 raft_consensus.cc:1226] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 8. Ops: []
I20240507 08:06:39.648573 13282 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.702013 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1691) took 1381 ms. Trace:
I20240507 08:06:39.705984 10335 rpcz_store.cc:276] 0507 08:06:38.320558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.626119 (+1305561us) service_pool.cc:225] Handling call
0507 08:06:39.636506 (+ 10387us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.701794 (+ 65288us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.701964 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.696909 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1958) took 1540 ms. Trace:
I20240507 08:06:39.702486 10087 catalog_manager.cc:5484] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 8 to 9, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 9 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:39.706934 10265 rpcz_store.cc:276] 0507 08:06:38.156714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.156966 (+ 252us) service_pool.cc:225] Handling call
0507 08:06:38.157336 (+ 370us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:38.341382 (+184046us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:38.341417 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:38.341862 (+ 445us) log.cc:842] Serialized 26 byte log entry
0507 08:06:39.515405 (+1173543us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:39.515442 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.594182 (+ 78740us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54002be654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:39.594259 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.686646 (+ 92387us) raft_consensus.cc:1671] finished
0507 08:06:39.686674 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.696847 (+ 10173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":442,"raft.queue_time_us":223,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6008,"spinlock_wait_cycles":422677376,"thread_start_us":1158866,"threads_started":1,"wal-append.queue_time_us":1235530}
I20240507 08:06:39.707741 10087 catalog_manager.cc:5484] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 4, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 4 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:39.708992 10087 catalog_manager.cc:5484] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 9 to 10, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 10 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:39.686264 13325 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.711512 10087 catalog_manager.cc:5484] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 3 to 4, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 4 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:39.712754 10087 catalog_manager.cc:5484] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:39.713737 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2214) took 1816 ms. Trace:
I20240507 08:06:39.693274 13152 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.715737 10087 catalog_manager.cc:5484] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 7 to 8, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 8 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:39.714756 10191 rpcz_store.cc:276] 0507 08:06:37.897702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.901048 (+ 3346us) service_pool.cc:225] Handling call
0507 08:06:37.901304 (+ 256us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.428740 (+1527436us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.713498 (+284758us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54001b3d54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:39.713666 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3210435968}
I20240507 08:06:39.692893 13361 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:39.724895 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1837) took 1406 ms. Trace:
W20240507 08:06:39.725193 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 392184us
I20240507 08:06:39.726121 10087 catalog_manager.cc:5484] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 8 to 9, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 9 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: HEALTHY } } }
I20240507 08:06:39.723070 13325 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.726150 10191 rpcz_store.cc:276] 0507 08:06:38.318024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.724191 (+1406167us) service_pool.cc:225] Handling call
0507 08:06:39.724456 (+ 265us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.724637 (+ 181us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.724661 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.724701 (+ 40us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:39.724731 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.724771 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.724856 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.741426 13332 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:39.729440 13398 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20240507 08:06:39.730275 13377 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.730913 12979 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.730952 13337 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.731714 13319 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.732344 13168 leader_election.cc:288] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.650343 13205 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.744491 13205 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.745085 13205 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.709913 13306 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.733469 13316 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.702621 13147 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:39.733326 13361 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.735725 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1831) took 1603 ms. Trace:
I20240507 08:06:39.699352 13169 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.730597 13399 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.726418 13152 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:39.735745 13289 consensus_meta.cc:287] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 1.407s user 0.010s sys 0.001s
I20240507 08:06:39.742079 13403 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.738044 13141 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.738708 13280 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.739089 13400 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:39.739500 13310 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:39.739689 13366 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 1.452s user 0.000s sys 0.007s
I20240507 08:06:39.774605 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 409) took 2228 ms. Trace:
I20240507 08:06:39.739774 13329 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.695103 13322 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.739848 13336 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.783937 13361 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.741521 10191 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:39.784994 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1838) took 1456 ms. Trace:
I20240507 08:06:39.753854 13412 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.705209 13404 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.746259 13306 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.747885 13157 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.654608 13305 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.776652 13400 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.770711 13403 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.867942 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:39.802290 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 248181us
I20240507 08:06:39.886570 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1693) took 1565 ms. Trace:
I20240507 08:06:39.808560 10332 raft_consensus.cc:1226] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:39.803889 10156 leader_election.cc:398] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10.
I20240507 08:06:39.892551 10156 leader_election.cc:401] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20240507 08:06:39.749223 10190 rpcz_store.cc:276] 0507 08:06:38.132400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.132627 (+ 227us) service_pool.cc:225] Handling call
0507 08:06:38.132945 (+ 318us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:38.143797 (+ 10852us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:38.143826 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:38.144223 (+ 397us) log.cc:842] Serialized 26 byte log entry
0507 08:06:39.243479 (+1099256us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:39.243503 (+ 24us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.243554 (+ 51us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:39.735532 (+491978us) raft_consensus.cc:1671] finished
0507 08:06:39.735551 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.735671 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2084,"raft.queue_time_us":12502,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":113114,"thread_start_us":939767,"threads_started":1,"wal-append.queue_time_us":1025335}
I20240507 08:06:39.737721 13198 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.753640 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2227) took 1462 ms. Trace:
I20240507 08:06:39.755098 10262 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:39.703873 13327 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:39.754022 13346 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.930s user 0.020s sys 0.016s
I20240507 08:06:39.770387 12998 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:39.770660 13086 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.776844 10158 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:39.745512 13205 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.779683 13318 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:39.775075 10340 rpcz_store.cc:276] 0507 08:06:37.546077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.546342 (+ 265us) service_pool.cc:225] Handling call
0507 08:06:37.546710 (+ 368us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.551634 (+ 4924us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.551682 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.568487 (+ 16805us) log.cc:842] Serialized 26 byte log entry
0507 08:06:37.981095 (+412608us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:37.981133 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:37.981214 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:38.308928 (+327714us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b1800348e20. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8defd1b8 00007fcf8df00691 00007fcf8df00649 00007fcf8df005d9 00007fcf8df005a1 00007fcf8deff89d 00007fcf97164224 00007fcf97164059 00007fcf8dedf936
0507 08:06:39.611681 (+1302753us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:39.774329 (+162648us) raft_consensus.cc:1671] finished
0507 08:06:39.774362 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.774514 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4476,"raft.queue_time_us":27595,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"spinlock_wait_cycles":72595584,"thread_start_us":402699,"threads_started":1,"wal-append.queue_time_us":709966}
W20240507 08:06:39.782336 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1695) took 1454 ms (client timeout 1732 ms). Trace:
I20240507 08:06:39.782384 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.785514 10191 rpcz_store.cc:276] 0507 08:06:38.329013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.740913 (+1411900us) service_pool.cc:225] Handling call
0507 08:06:39.741209 (+ 296us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.784759 (+ 43550us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.784942 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:39.728432 10087 catalog_manager.cc:5484] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 5 to 6, leader changed from 5d6656a616b747d393ff3de5328d2540 (127.8.213.132) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240507 08:06:39.749562 10187 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:39.935333 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:39.811213 13141 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.784407 13336 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.796955 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:39.802502 10231 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.003s sys 0.000s
W20240507 08:06:39.810252 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 316581us
W20240507 08:06:39.940474 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 163761us
I20240507 08:06:39.833154 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.869450 13322 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.799011 13418 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:39.967653 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:39.968353 10266 raft_consensus.cc:2354] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10.
I20240507 08:06:39.797996 13280 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.821313 10943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.830181 13417 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.797544 13409 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.950143 10087 catalog_manager.cc:5484] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
W20240507 08:06:39.948503 10333 rpcz_store.cc:269] 0507 08:06:38.327703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.356155 (+ 28452us) service_pool.cc:225] Handling call
0507 08:06:39.782169 (+1426014us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b540014f0d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:39.782286 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":803777408}
I20240507 08:06:39.838883 10908 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:40.014590 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:40.014945 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 176265us
I20240507 08:06:39.953869 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:39.778525 13310 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:39.786213 13329 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.800135 13276 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.889112 10339 rpcz_store.cc:276] 0507 08:06:38.321164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.490710 (+169546us) service_pool.cc:225] Handling call
0507 08:06:39.886364 (+1395654us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b5400049f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:39.886498 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":886238464}
I20240507 08:06:39.804091 13407 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.813603 13289 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.886626 13152 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.800375 13361 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:39.956719 10231 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.669s user 0.011s sys 0.006s
I20240507 08:06:39.948029 13061 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20240507 08:06:39.869028 13157 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.063767 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 385) took 1769 ms. Trace:
I20240507 08:06:40.049754 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:40.046763 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 220759us
I20240507 08:06:39.937029 13399 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:39.964712 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.965482 10263 raft_consensus.cc:1261] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20240507 08:06:39.826974 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.937496 10183 rpcz_store.cc:276] 0507 08:06:38.291565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.569259 (+1277694us) service_pool.cc:225] Handling call
0507 08:06:39.589643 (+ 20384us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.636612 (+ 46969us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.636640 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.636677 (+ 37us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:39.636712 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:39.753427 (+116715us) spinlock_profiling.cc:243] Waited 46.8 ms on lock 0x7b54002d7654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:39.753468 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:39.753569 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107663616}
I20240507 08:06:39.938463 10306 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:39.970079 10266 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20240507 08:06:39.976122 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:39.935711 13367 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:39.870129 13375 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:39.811208 10277 tablet_copy_service.cc:342] P bcf08c8276ba4a9ba646ad0f7090fcd8: Request end of tablet copy session 5d6656a616b747d393ff3de5328d2540-0c898839b7c84ce6ba0e135f778ad72d received from {username='slave'} at 127.0.0.1:39118
I20240507 08:06:39.823402 13117 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:39.940078 13169 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.009536 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:39.847662 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.012220 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1970) took 1061 ms. Trace:
I20240507 08:06:39.828142 13377 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.140s
I20240507 08:06:39.847457 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.854277 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.952793 13426 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 10
I20240507 08:06:40.101657 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 386) took 1806 ms. Trace:
I20240507 08:06:40.102017 10184 rpcz_store.cc:276] 0507 08:06:38.294746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.701211 (+1406465us) service_pool.cc:225] Handling call
0507 08:06:39.721241 (+ 20030us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.745809 (+ 24568us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.745840 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.745878 (+ 38us) raft_consensus.cc:1631] Marking committed up to 9
0507 08:06:39.745908 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.101414 (+355506us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b54002dbed4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:40.101458 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.101585 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":56110976}
I20240507 08:06:40.029668 10262 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:40.103247 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:40.103534 10262 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1969) took 1256 ms. Trace:
I20240507 08:06:40.103916 10184 raft_consensus.cc:2379] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 7: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:40.103868 10262 rpcz_store.cc:276] 0507 08:06:38.846906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.847040 (+ 134us) service_pool.cc:225] Handling call
0507 08:06:40.103402 (+1256362us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54002be154. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:40.103489 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1168321920}
I20240507 08:06:40.104712 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.104859 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.106024 13205 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.029940 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 384) took 1735 ms. Trace:
I20240507 08:06:39.875939 13364 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:40.107084 13387 log.cc:924] Time spent T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.812s user 0.003s sys 0.006s
I20240507 08:06:40.108521 13364 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:39.950659 13416 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:40.109230 13416 raft_consensus.cc:686] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:40.110121 13416 consensus_queue.cc:237] T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.090695 10277 tablet_copy_service.cc:433] P bcf08c8276ba4a9ba646ad0f7090fcd8: ending tablet copy session 5d6656a616b747d393ff3de5328d2540-0c898839b7c84ce6ba0e135f778ad72d on tablet 0c898839b7c84ce6ba0e135f778ad72d with peer 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:40.113761 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2221) took 2213 ms. Trace:
I20240507 08:06:40.036243 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:39.958525 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:39.880268 13147 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.041707 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.088114 12979 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:40.047472 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.055017 10155 leader_election.cc:398] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10.
I20240507 08:06:39.954612 13086 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:39.950736 12998 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.066351 10188 rpcz_store.cc:276] 0507 08:06:38.294496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.690102 (+1395606us) service_pool.cc:225] Handling call
0507 08:06:39.704364 (+ 14262us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.704534 (+ 170us) raft_consensus.cc:1520] Early marking committed up to index 7
0507 08:06:39.961602 (+257068us) log.cc:842] Serialized 14 byte log entry
0507 08:06:40.062442 (+100840us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.062533 (+ 91us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:40.062657 (+ 124us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.062796 (+ 139us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.063284 (+ 488us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3334,"prepare.queue_time_us":135794,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"thread_start_us":336569,"threads_started":2}
W20240507 08:06:40.087245 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 148903us
W20240507 08:06:40.160851 10259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1968) took 1581 ms (client timeout 1640 ms). Trace:
W20240507 08:06:40.088217 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 112217us
I20240507 08:06:40.161409 10306 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.089120 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:39.968179 13141 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.169031 13086 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.169255 13141 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.170513 13415 raft_consensus.cc:2781] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Leader election won for term 11
I20240507 08:06:40.170892 13403 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.171020 13415 raft_consensus.cc:686] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:40.171918 13415 consensus_queue.cc:237] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.173144 13327 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.061908 13152 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:40.173835 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.173694 12979 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.091228 10258 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
W20240507 08:06:40.161278 10259 rpcz_store.cc:269] 0507 08:06:38.579911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.627608 (+ 47697us) service_pool.cc:225] Handling call
0507 08:06:40.127950 (+1500342us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b5400116054. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:40.160795 (+ 32845us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":778185472}
I20240507 08:06:40.007894 13336 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20240507 08:06:39.887475 13404 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.179142 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2129) took 2285 ms. Trace:
I20240507 08:06:39.881049 13412 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.097474 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.097973 10231 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.045221 13413 raft_consensus.cc:2781] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:40.105974 13290 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.033438 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.027097 13280 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.198168 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2226) took 1925 ms. Trace:
I20240507 08:06:40.093745 10264 rpcz_store.cc:276] 0507 08:06:38.983963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.984559 (+ 596us) service_pool.cc:225] Handling call
0507 08:06:39.232279 (+247720us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.232516 (+ 237us) raft_consensus.cc:1520] Early marking committed up to index 6
0507 08:06:39.724731 (+492215us) log.cc:842] Serialized 14 byte log entry
0507 08:06:40.011781 (+287050us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.011854 (+ 73us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:40.011913 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.011999 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.012144 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":494467,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":396,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":9761,"thread_start_us":715998,"threads_started":2}
I20240507 08:06:40.105374 10262 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 6.
I20240507 08:06:40.034135 13409 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.119395 13274 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:40.120311 13306 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.161408 13169 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:40.121246 13305 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.086674 13399 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.045082 13407 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.050297 13289 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.046010 13276 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.129392 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.182933 13325 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:39.935983 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 132293us
I20240507 08:06:40.114109 10186 rpcz_store.cc:276] 0507 08:06:37.900433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:37.903256 (+ 2823us) service_pool.cc:225] Handling call
0507 08:06:37.903500 (+ 244us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:37.903795 (+ 295us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:37.903820 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:37.904164 (+ 344us) log.cc:842] Serialized 26 byte log entry
0507 08:06:38.241491 (+337327us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:38.241548 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:38.241665 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:40.106541 (+1864876us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:40.113413 (+ 6872us) raft_consensus.cc:1671] finished
0507 08:06:40.113434 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.113688 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":74,"spinlock_wait_cycles":18281856,"thread_start_us":336783,"threads_started":1,"wal-append.queue_time_us":1279919}
I20240507 08:06:40.211135 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:40.160390 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 144884us
I20240507 08:06:40.162557 10266 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
W20240507 08:06:40.230252 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 180909us
W20240507 08:06:40.163820 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 109044us
I20240507 08:06:40.165064 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:40.167399 13147 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:40.093904 13168 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:40.096009 13319 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:40.242153 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.243275 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.175050 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.176982 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.178514 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1834) took 1979 ms. Trace:
I20240507 08:06:40.245868 10185 rpcz_store.cc:276] 0507 08:06:38.199520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.548177 (+1348657us) service_pool.cc:225] Handling call
0507 08:06:39.551640 (+ 3463us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.699926 (+148286us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:39.699956 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:39.699995 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:39.700021 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.178261 (+478240us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000168d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:40.178322 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.178452 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340788096}
I20240507 08:06:40.247136 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:40.248175 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.249606 10261 raft_consensus.cc:1261] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:40.178571 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.175465 10304 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.095886 13198 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:40.179512 10332 rpcz_store.cc:276] 0507 08:06:37.894166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.272008 (+1377842us) service_pool.cc:225] Handling call
0507 08:06:39.272296 (+ 288us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.891292 (+618996us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.178939 (+287647us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b540014dcd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:40.179080 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1233315712}
I20240507 08:06:40.005798 13418 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.180207 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:40.251705 10260 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:40.175420 10232 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.182332 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.061982 13361 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:40.185729 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 87853.9us
W20240507 08:06:40.188179 10187 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1832) took 2052 ms (client timeout 1732 ms). Trace:
I20240507 08:06:40.116150 13406 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:40.208321 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.208381 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.043550 13417 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.170922 12998 leader_election.cc:288] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.098498 13117 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.230232 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.106684 10192 rpcz_store.cc:276] 0507 08:06:38.294274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.640034 (+1345760us) service_pool.cc:225] Handling call
0507 08:06:39.640297 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.640431 (+ 134us) raft_consensus.cc:1520] Early marking committed up to index 10
0507 08:06:39.808898 (+168467us) log.cc:842] Serialized 14 byte log entry
0507 08:06:40.029619 (+220721us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.029673 (+ 54us) raft_consensus.cc:1631] Marking committed up to 10
0507 08:06:40.029719 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.029781 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.029890 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50911,"prepare.queue_time_us":199690,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":55148,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":23673,"thread_start_us":338086,"threads_started":2,"wal-append.queue_time_us":263051}
I20240507 08:06:40.261847 10907 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:40.186236 13413 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:40.230013 13205 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:40.232564 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.234341 10265 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:40.233505 10155 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.173830 13152 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.235790 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.237449 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:40.275530 10262 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:40.240175 10156 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.273169 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.196310 13403 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240507 08:06:40.252419 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 77143.5us
I20240507 08:06:40.253731 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:40.279240 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 45842.4us
I20240507 08:06:40.280112 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.285365 10232 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:40.280747 10264 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:40.288249 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.253744 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.263048 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:40.255885 10187 rpcz_store.cc:269] 0507 08:06:38.135566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:38.371964 (+236398us) service_pool.cc:225] Handling call
0507 08:06:40.187969 (+1816005us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54003b4554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:40.188096 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1145816832}
I20240507 08:06:40.162791 10306 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:40.291663 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 128999us
W20240507 08:06:40.263626 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 88297.9us
I20240507 08:06:40.280268 10188 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:40.293279 10304 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.219528 13169 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.303822 10260 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:40.305014 10187 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "88a26dac37fa4243b8c2b3d47f403488"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:40.305928 10187 raft_consensus.cc:595] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Received request to transfer leadership to TS 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:40.306418 10187 raft_consensus.cc:603] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:40.307224 10185 raft_consensus.cc:2379] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:40.308549 10185 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:40.274197 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.294601 10191 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:40.294776 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:40.265235 13325 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:40.313877 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 387) took 1824 ms. Trace:
I20240507 08:06:40.314394 10186 rpcz_store.cc:276] 0507 08:06:38.489158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.312772 (+1823614us) service_pool.cc:225] Handling call
0507 08:06:40.313146 (+ 374us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:40.313361 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:40.313397 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.313452 (+ 55us) raft_consensus.cc:1631] Marking committed up to 10
0507 08:06:40.313496 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.313668 (+ 172us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.313806 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:40.246748 13289 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.266286 13319 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:40.322961 10184 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:40.250697 13400 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.314095 10258 raft_consensus.cc:2379] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 7: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:40.295490 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:40.317116 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 77046.3us
I20240507 08:06:40.272135 13147 raft_consensus.cc:2781] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:40.268554 13310 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.342164 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:40.328747 10262 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:40.330639 10266 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:40.250607 13157 consensus_queue.cc:1035] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20240507 08:06:40.211545 13367 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.268465 13413 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:40.281771 13403 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:40.354990 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.343228 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.398635 10264 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 5.
I20240507 08:06:40.293015 13121 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:40.289489 13316 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:40.355219 10186 raft_consensus.cc:1261] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20240507 08:06:40.395627 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 66157.8us
I20240507 08:06:40.250856 13168 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.288120 13329 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.268903 13336 consensus_queue.cc:1035] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
I20240507 08:06:40.326649 13325 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:40.282935 13205 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.323253 13279 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:40.465939 10155 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.336534 13375 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:40.322127 13198 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
W20240507 08:06:40.458904 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 157921us
W20240507 08:06:40.340101 13428 log.cc:924] Time spent T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:06:40.266901 13406 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.475800 13375 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:40.482746 13168 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.482854 13375 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:40.483050 13406 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.483390 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.483904 10259 raft_consensus.cc:1226] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20240507 08:06:40.469177 13325 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:40.487221 13421 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:40.487634 13421 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:40.488483 13421 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.341360 13418 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:40.395468 13435 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20240507 08:06:40.453366 13426 raft_consensus.cc:2726] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10.
I20240507 08:06:40.461512 10185 raft_consensus.cc:1226] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:40.463677 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.351464 13332 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:40.341637 13147 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
W20240507 08:06:40.402304 13366 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20240507 08:06:40.467780 10260 raft_consensus.cc:1226] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:40.349462 13417 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.347949 12979 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.350514 13409 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.465170 10263 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:40.347154 13361 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.348541 13412 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.342628 13152 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.345551 13276 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.476655 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:40.476826 10192 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240507 08:06:40.403900 13403 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:40.346246 13280 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:40.483073 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:40.500363 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 200033us
I20240507 08:06:40.392787 13289 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20240507 08:06:40.459715 13117 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.509802 10183 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:40.402395 13374 consensus_queue.cc:1035] T b909813a654d41cf88fb9fee2f8006bc P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.096s
I20240507 08:06:40.479275 13414 raft_consensus.cc:2726] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:40.482738 13169 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.571285 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.604143 10191 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
W20240507 08:06:40.584178 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 258034us
I20240507 08:06:40.512049 13421 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:40.510919 13147 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:40.585137 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.563869 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:40.596377 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 119824us
I20240507 08:06:40.503361 13198 raft_consensus.cc:686] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:40.644054 13198 consensus_queue.cc:237] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:40.607335 10908 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:40.513568 13332 raft_consensus.cc:686] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:40.645701 13429 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:40.646274 13332 consensus_queue.cc:237] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.646368 13429 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.563282 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.610368 10192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
W20240507 08:06:40.659111 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 50446.1us
I20240507 08:06:40.617388 10155 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.513835 13274 consensus_queue.cc:1035] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20240507 08:06:40.600719 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.594444 13404 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:40.198720 10189 rpcz_store.cc:276] 0507 08:06:38.272790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.533255 (+1260465us) service_pool.cc:225] Handling call
0507 08:06:39.573719 (+ 40464us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:39.573904 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 7
0507 08:06:40.060384 (+486480us) log.cc:842] Serialized 14 byte log entry
0507 08:06:40.197768 (+137384us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.197833 (+ 65us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:40.197889 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.197967 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:40.198103 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":256713,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":9226,"raft.queue_time_us":112,"thread_start_us":607139,"threads_started":2}
I20240507 08:06:40.626797 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:55810 (request call id 530) took 2296 ms. Trace:
I20240507 08:06:40.628337 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.600824 13310 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:40.659252 10192 raft_consensus.cc:595] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.680941 10192 raft_consensus.cc:603] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:40.610389 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.662992 13461 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:40.660131 13086 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:40.679319 10185 rpcz_store.cc:276] 0507 08:06:38.330083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.625487 (+2295404us) service_pool.cc:225] Handling call
0507 08:06:40.626722 (+ 1235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:40.521713 13319 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.683946 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.703974 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.704773 10262 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:40.705333 10155 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.705807 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.689119 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.689981 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.690788 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.708860 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.691809 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.709983 10341 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:40.692798 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.699426 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.711373 10336 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:40.713733 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:40.714304 10337 raft_consensus.cc:3037] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:40.672621 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:40.716431 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.677922 13157 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:40.718344 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:40.717832 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 45391.5us
I20240507 08:06:40.719429 13460 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240507 08:06:40.720079 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e57f967aaaf34bd5acd85d28edeb4abe, TS 88a26dac37fa4243b8c2b3d47f403488 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.719959 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.721099 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:40.757073 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e57f967aaaf34bd5acd85d28edeb4abe, TS 88a26dac37fa4243b8c2b3d47f403488 -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.757532 10263 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:40.721889 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.733856 13406 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:40.683226 13310 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:40.730046 13325 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.745855 13423 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:40.703438 13426 consensus_queue.cc:1046] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
W20240507 08:06:40.749900 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 440402us
I20240507 08:06:40.698298 13461 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.752012 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.753789 10265 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:40.750394 10155 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:40.770213 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 268370us
I20240507 08:06:40.770759 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.763320 10186 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db7c60767d74458094c1d86a49e87980"
dest_uuid: "59382e6647c549d89d83c70233c0667e"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:51764
I20240507 08:06:40.771157 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.771595 10186 raft_consensus.cc:595] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.771991 10186 raft_consensus.cc:603] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240507 08:06:40.764319 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.767668 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.720618 13367 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:40.776721 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.772586 10258 raft_consensus.cc:2454] T 7b89492a3abf415da273cc9a5b1227bf P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 4.
I20240507 08:06:40.782377 10335 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:40.777810 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.701447 13086 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.780153 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.783980 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.786090 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.786500 10185 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:40.786676 10340 raft_consensus.cc:2454] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:40.698879 13411 consensus_queue.cc:1035] T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
I20240507 08:06:40.796149 10260 raft_consensus.cc:2379] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 7: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:40.783514 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.778986 10232 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:40.790930 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.794447 10263 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:40.757726 13460 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:40.770015 13310 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:40.797322 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.799419 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.806895 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.807891 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.808686 10188 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 3.
I20240507 08:06:40.808485 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:40.810699 10232 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:40.811612 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.812721 10185 raft_consensus.cc:3037] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:40.799753 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.800676 10265 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:40.802606 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.795789 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.804231 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffd4c65b5ad04f48916c3e359f7e45c9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.806764 10155 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.774669 13429 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:40.814888 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:40.816814 10155 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.812438 10183 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
W20240507 08:06:40.801800 13452 log.cc:924] Time spent T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20240507 08:06:40.785785 13426 consensus_queue.cc:1225] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer 88a26dac37fa4243b8c2b3d47f403488 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240507 08:06:40.821952 10232 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.823527 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.785018 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.806572 10937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.769531 13406 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.797673 13061 raft_consensus.cc:2726] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:40.801944 13409 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:40.824156 10155 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:40.803896 13460 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:40.806088 13346 consensus_queue.cc:1046] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20240507 08:06:40.833595 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.825421 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:40.843024 13325 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.801932 13205 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:40.845005 13413 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:40.774338 13375 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.846227 13374 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:40.858285 10185 raft_consensus.cc:2454] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:40.859156 13086 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:40.784519 13280 consensus_queue.cc:1035] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
W20240507 08:06:40.860910 13456 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20240507 08:06:40.831285 9046 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
W20240507 08:06:40.862293 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 271053us
W20240507 08:06:40.906806 10304 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.732s (SENT) [suppressed 6 similar messages]
I20240507 08:06:40.832621 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:40.816154 13169 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20240507 08:06:40.889894 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 65821.2us
I20240507 08:06:40.947495 10184 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:40.861822 13441 consensus_queue.cc:1046] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20240507 08:06:40.901229 13117 raft_consensus.cc:2781] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:40.895915 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:40.813598 13464 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:40.825366 13361 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240507 08:06:40.897439 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 76667.6us
I20240507 08:06:40.830791 13152 raft_consensus.cc:2781] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:40.903123 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.822551 13417 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.073s
I20240507 08:06:40.908552 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.978489 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:40.908838 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.909415 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.910033 10338 raft_consensus.cc:1261] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240507 08:06:40.943207 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.980607 10336 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 9.
I20240507 08:06:40.944200 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.908253 9046 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240507 08:06:40.862306 13453 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:40.911456 10340 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:40.895156 13367 raft_consensus.cc:2781] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240507 08:06:40.962478 10934 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
W20240507 08:06:40.897629 13415 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.027s sys 0.031s
W20240507 08:06:40.998883 13415 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s user 0.027s sys 0.031s
I20240507 08:06:41.002250 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.828001 10263 raft_consensus.cc:2454] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:40.904093 13426 raft_consensus.cc:3032] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Stepping down as leader of term 9
W20240507 08:06:40.961527 13448 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.622s user 0.004s sys 0.000s
I20240507 08:06:40.896556 13409 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:40.975517 10189 raft_consensus.cc:2396] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20240507 08:06:40.910224 13374 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:40.984380 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.026566 10338 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:41.016587 10188 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:41.027372 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.986833 10333 raft_consensus.cc:2379] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 6: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:40.987435 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e6f917a0bed4f6289c1abc13f6260b9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.028189 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.030990 10263 raft_consensus.cc:1261] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:41.033185 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.028018 10338 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 5.
I20240507 08:06:40.989990 10306 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:40.989250 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.035799 10340 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:41.036583 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.037169 10340 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:40.991716 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:40.993515 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:40.997761 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.008307 13276 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:41.009173 10304 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:40.995160 13367 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:40.970196 13117 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:41.013278 10155 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:41.072161 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 59013.3us
I20240507 08:06:41.014866 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.008764 13445 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:41.038283 10341 raft_consensus.cc:2379] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 3: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:41.010509 13426 raft_consensus.cc:729] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:41.040180 13418 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:41.011657 10156 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.042382 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:41.084782 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 73250.5us
I20240507 08:06:41.042534 10304 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.043031 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:41.087038 10338 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:41.030005 13417 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:41.043855 13169 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:41.097295 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.098217 10332 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 9: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:41.049800 13406 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.016431 13152 raft_consensus.cc:686] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:41.079185 13117 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:41.050482 13461 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:41.052179 13465 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
I20240507 08:06:41.016295 13361 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.052740 13415 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:41.053517 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.088196 12979 raft_consensus.cc:2741] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240507 08:06:41.060783 13423 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:41.063789 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:41.147738 10339 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:41.147868 10333 raft_consensus.cc:2454] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3.
I20240507 08:06:41.079537 13276 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.068619 13399 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.069505 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.161669 13276 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.141906 13375 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.162302 10336 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:41.162619 13152 consensus_queue.cc:237] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.054735 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.075699 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:41.038506 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 48618.6us
W20240507 08:06:41.083652 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 68900.6us
W20240507 08:06:41.089051 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 46618.5us
I20240507 08:06:41.167351 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.167500 10304 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.168392 10304 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.117676 13169 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.171131 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:41.172266 10306 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:41.058153 13457 log.cc:924] Time spent T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20240507 08:06:41.057929 13157 raft_consensus.cc:3032] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:41.089444 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.214942 10341 raft_consensus.cc:2379] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:41.055555 13280 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:41.217185 13280 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:41.217527 10339 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:41.080159 13367 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:41.114616 10338 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:41.220049 10341 raft_consensus.cc:1226] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:41.081378 13442 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:41.054273 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.053764 13403 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.090440 13346 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:41.090837 13322 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:41.032076 13464 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.104s
I20240507 08:06:41.151367 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:41.066006 10337 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:41.155503 13459 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:41.147866 13426 consensus_queue.cc:260] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.158165 13147 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.117504 13417 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.084084 13445 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.173205 10306 leader_election.cc:302] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:41.153213 13423 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:41.175197 13332 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240507 08:06:41.178359 13461 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:41.178436 13421 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:41.180039 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.187165 13169 leader_election.cc:288] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.141563 13361 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:41.271381 13310 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.169413 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.217643 13280 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:41.277941 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 507) took 1181 ms. Trace:
W20240507 08:06:41.278159 13478 log.cc:924] Time spent T 00000000000000000000000000000000 P 9ee8354230fc476f93816bfbab26bf0a: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20240507 08:06:41.278322 10266 rpcz_store.cc:276] 0507 08:06:40.096155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.397645 (+301490us) service_pool.cc:225] Handling call
0507 08:06:40.629818 (+232173us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:40.630026 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:40.630064 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.630119 (+ 55us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:40.630162 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:40.630232 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:41.277787 (+647555us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b540027ea50. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:41.277887 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":533546368}
I20240507 08:06:41.279857 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:41.001684 13453 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:41.280531 10266 raft_consensus.cc:2354] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:41.281993 13453 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:41.221552 13442 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.283772 13442 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.090994 13418 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:41.285668 13418 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:41.225036 10232 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
W20240507 08:06:41.286319 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 61417.8us
I20240507 08:06:41.286926 13403 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.287657 13403 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:41.288044 13403 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:41.288461 13403 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:41.230844 10258 raft_consensus.cc:2354] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for earlier term 6. Current term is 7.
I20240507 08:06:41.216472 13415 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:41.291821 13415 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.292161 13415 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:41.292976 13417 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.224234 13409 consensus_queue.cc:237] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.294479 10155 leader_election.cc:398] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:41.294797 10155 leader_election.cc:401] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240507 08:06:41.180058 13413 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.248920 10339 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240507 08:06:41.223484 13336 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:41.214594 13157 raft_consensus.cc:729] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:41.254169 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 81077us
I20240507 08:06:41.309161 10306 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:41.239480 13414 raft_consensus.cc:2781] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240507 08:06:41.260634 10260 raft_consensus.cc:2379] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 9: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:41.256592 13412 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:41.275738 13117 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:41.276199 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 106875us
I20240507 08:06:41.281268 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.327673 10266 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:41.328150 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.319216 10306 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:41.273437 13310 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.255002 13322 raft_consensus.cc:2726] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:41.256052 13332 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.262620 13426 raft_consensus.cc:3037] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:41.329046 10304 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.330173 10155 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.247180 13459 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:41.331806 10306 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:41.276391 13086 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:41.332484 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:41.300860 13418 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:41.332851 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.300690 13336 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:41.371943 10304 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:41.325069 13461 raft_consensus.cc:2781] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:41.402772 10188 raft_consensus.cc:1226] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:41.334120 13364 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20240507 08:06:41.375461 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 43734.9us
I20240507 08:06:41.389312 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.337236 13332 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:41.408219 10266 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:41.337371 13475 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:41.386309 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:41.382831 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.383956 10156 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:41.419723 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 47911.3us
I20240507 08:06:41.416486 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.425678 13086 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:41.417029 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:41.427043 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 96987.3us
I20240507 08:06:41.418107 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.428440 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.424232 10191 raft_consensus.cc:2362] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 3. Re-sending same reply.
I20240507 08:06:41.416536 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.429388 10188 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 5.
I20240507 08:06:41.429520 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b909813a654d41cf88fb9fee2f8006bc" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.425170 10264 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:41.430187 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:41.383910 13322 raft_consensus.cc:2726] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:41.402511 13319 raft_consensus.cc:2741] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20240507 08:06:41.409950 13336 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:41.426112 13086 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:41.416110 13364 raft_consensus.cc:482] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.437131 10184 raft_consensus.cc:2379] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 8: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:41.330785 13412 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.458946 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.453341 10260 raft_consensus.cc:1226] T 82c5eb4243284619ad55e8a724ff32de P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20240507 08:06:41.432766 13460 consensus_meta.cc:287] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 0.534s user 0.010s sys 0.001s
I20240507 08:06:41.435593 13375 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240507 08:06:41.438688 10263 raft_consensus.cc:1261] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240507 08:06:41.467419 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.459882 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.460407 10304 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.462658 13117 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.463394 13361 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:41.432276 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 48417.3us
I20240507 08:06:41.480984 10156 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.472090 10266 raft_consensus.cc:2454] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:41.482095 10156 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:41.458947 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.445379 13276 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:41.471159 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:41.458294 13364 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:41.453171 13086 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:41.483111 10156 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:41.402729 13157 consensus_queue.cc:260] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.497467 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 501) took 1685 ms. Trace:
I20240507 08:06:41.477771 13121 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:41.498215 10261 rpcz_store.cc:276] 0507 08:06:39.811583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.813788 (+ 2205us) service_pool.cc:225] Handling call
0507 08:06:39.814123 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:40.249946 (+435823us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:41.497235 (+1247289us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54002adad4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:41.497393 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1001197824}
I20240507 08:06:41.470618 13375 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.500658 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:41.503638 10156 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:41.493266 13416 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.151s user 0.010s sys 0.012s
I20240507 08:06:41.485395 13147 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:41.464185 13460 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:41.547837 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:41.491112 13276 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.512601 10156 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.489750 13319 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:41.556480 10186 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:41.505965 13441 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20240507 08:06:41.560145 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 47736.8us
I20240507 08:06:41.560348 13441 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:41.561754 10156 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:41.562433 10156 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.563191 10156 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.564167 10156 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.565132 10156 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:41.565981 13409 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:41.566385 13409 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.566727 13409 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
W20240507 08:06:41.555568 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 67832.1us
I20240507 08:06:41.498567 13469 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:41.500541 13310 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:41.587764 10266 raft_consensus.cc:3032] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 LEADER]: Stepping down as leader of term 10
I20240507 08:06:41.501660 13327 raft_consensus.cc:2781] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:41.510011 13375 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:41.609660 10908 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:41.545786 13416 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.204s user 0.014s sys 0.039s
I20240507 08:06:41.500191 13492 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:41.616142 10266 raft_consensus.cc:729] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:41.617055 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:41.513373 13445 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.514199 13346 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:41.579422 13276 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:41.617290 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.624254 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.627228 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.633098 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:41.634024 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 210105us
I20240507 08:06:41.587440 13469 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:41.633594 13278 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:41.508852 13198 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.116s
I20240507 08:06:41.639552 10184 raft_consensus.cc:2454] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 9.
I20240507 08:06:41.656124 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:41.509614 13121 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.618914 13327 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:06:41.621770 13456 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.074s user 0.000s sys 0.004s
I20240507 08:06:41.658859 10232 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:41.658879 10332 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:41.659984 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34da9500ed541909dbdbcc3be70a067" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.660259 10232 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.661059 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:41.661098 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:41.618005 13152 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:41.631458 13310 raft_consensus.cc:2726] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:41.667282 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:41.668007 10263 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:41.668982 10332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:41.683271 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:41.622112 13461 raft_consensus.cc:686] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:41.653506 13469 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:41.653426 13416 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:41.717730 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 56767.3us
I20240507 08:06:41.748848 10232 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:41.683995 10264 raft_consensus.cc:2362] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8. Re-sending same reply.
I20240507 08:06:41.665104 13326 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader election won for term 3
I20240507 08:06:41.630115 10266 consensus_queue.cc:260] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:41.736042 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 104358us
I20240507 08:06:41.776188 10231 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:41.738444 13423 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:41.662519 13417 raft_consensus.cc:2726] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:41.665887 13327 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:41.638236 13117 consensus_queue.cc:1035] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20240507 08:06:41.747169 13429 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:41.640058 13141 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:41.671869 13441 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:41.666818 13459 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:41.777114 10266 raft_consensus.cc:3037] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Advancing to term 11
W20240507 08:06:41.773427 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 208440us
I20240507 08:06:41.891953 10156 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:41.893507 10156 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:41.894923 10156 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:41.777459 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 310271us
I20240507 08:06:41.783596 10264 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20240507 08:06:41.821122 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 166019us
W20240507 08:06:41.864444 13496 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
W20240507 08:06:41.856277 13485 log.cc:924] Time spent T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.391s user 0.001s sys 0.000s
I20240507 08:06:41.865083 13412 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:41.855201 13461 consensus_queue.cc:237] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.787513 13429 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:41.822089 13141 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:41.774353 13416 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.817731 13205 raft_consensus.cc:2781] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election won for term 7
W20240507 08:06:41.867355 13482 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.382s user 0.000s sys 0.001s
W20240507 08:06:41.881366 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 246439us
I20240507 08:06:41.779906 13415 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.753227 13325 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
W20240507 08:06:41.872494 13280 consensus_meta.cc:287] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.595s user 0.006s sys 0.005s
I20240507 08:06:41.896864 10155 leader_election.cc:302] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:41.898356 10156 leader_election.cc:302] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:41.870175 13503 log.cc:924] Time spent T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20240507 08:06:41.857177 13095 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:41.856456 13483 log.cc:924] Time spent T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.391s user 0.000s sys 0.001s
W20240507 08:06:41.975723 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 227045us
I20240507 08:06:41.976819 13464 raft_consensus.cc:2726] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240507 08:06:41.882757 13499 log.cc:924] Time spent T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.208s user 0.000s sys 0.001s
W20240507 08:06:41.944736 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 168689us
I20240507 08:06:42.032402 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:42.017722 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 119528us
W20240507 08:06:42.018247 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 121564us
I20240507 08:06:41.978535 13429 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:41.923521 13399 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20240507 08:06:41.910544 13492 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:41.911787 13152 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20240507 08:06:41.926753 13367 consensus_meta.cc:287] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.708s user 0.009s sys 0.014s
W20240507 08:06:42.036427 13501 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.378s user 0.000s sys 0.001s
I20240507 08:06:41.979149 13141 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
W20240507 08:06:41.989459 13337 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.004s sys 0.012s
I20240507 08:06:42.026293 13459 raft_consensus.cc:2726] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.030697 13508 raft_consensus.cc:2781] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:42.043807 13524 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:42.044710 13498 log.cc:924] Time spent T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20240507 08:06:42.045653 10231 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:42.017736 13280 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:42.065095 13337 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.601s user 0.004s sys 0.012s
I20240507 08:06:41.927722 10938 raft_consensus.cc:1226] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:41.999235 13409 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:41.944906 13460 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.035835 13121 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.038813 13399 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:41.992090 13117 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.079968 10231 leader_election.cc:302] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.032157 13095 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:42.103667 13480 log.cc:924] Time spent T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.647s user 0.001s sys 0.000s
I20240507 08:06:42.104569 13095 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.080482 13512 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:42.105233 13512 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.105751 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2254) took 1302 ms. Trace:
I20240507 08:06:42.107340 13512 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.106163 10185 rpcz_store.cc:276] 0507 08:06:40.809769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.917299 (+107530us) service_pool.cc:225] Handling call
0507 08:06:40.958657 (+ 41358us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:40.970459 (+ 11802us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:40.970501 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:40.971669 (+ 1168us) log.cc:842] Serialized 26 byte log entry
0507 08:06:41.239917 (+268248us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:41.239956 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:41.240015 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:42.103012 (+862997us) spinlock_profiling.cc:243] Waited 4.22 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:42.105435 (+ 2423us) raft_consensus.cc:1671] finished
0507 08:06:42.105457 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.105680 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11025,"raft.queue_time_us":14078,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":12086,"spinlock_wait_cycles":9716480,"thread_start_us":267893,"threads_started":1,"wal-append.queue_time_us":445971}
I20240507 08:06:42.100095 10340 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
W20240507 08:06:42.082156 13516 log.cc:924] Time spent T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240507 08:06:42.082299 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2251) took 1292 ms. Trace:
I20240507 08:06:42.109309 10186 rpcz_store.cc:276] 0507 08:06:40.790128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.831815 (+ 41687us) service_pool.cc:225] Handling call
0507 08:06:40.832165 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:41.560120 (+727955us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.082069 (+521949us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540005ee54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:42.082244 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1649154816}
W20240507 08:06:42.109946 13514 log.cc:924] Time spent T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20240507 08:06:42.110495 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:42.111076 13502 log.cc:924] Time spent T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.331s user 0.001s sys 0.000s
I20240507 08:06:42.111137 10186 raft_consensus.cc:2454] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:42.112171 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
W20240507 08:06:42.112903 13086 consensus_meta.cc:287] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.000s
I20240507 08:06:42.114558 10231 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.115327 10908 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:42.116149 10231 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.087517 13416 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.087917 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:42.089900 13508 raft_consensus.cc:686] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 10 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:41.777360 13169 consensus_queue.cc:1159] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "59382e6647c549d89d83c70233c0667e" responder_term: 11 status { last_received { term: 9 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 10. Current term is 11. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240507 08:06:42.048525 13423 leader_election.cc:288] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:41.985347 13510 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.134s
I20240507 08:06:41.923632 13310 consensus_queue.cc:1046] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
W20240507 08:06:42.078675 13403 consensus_meta.cc:287] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.789s user 0.008s sys 0.004s
I20240507 08:06:42.097051 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 2000) took 2125 ms. Trace:
I20240507 08:06:42.099416 10264 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:42.149785 10259 rpcz_store.cc:276] 0507 08:06:39.971446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.303980 (+332534us) service_pool.cc:225] Handling call
0507 08:06:40.304333 (+ 353us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:40.484257 (+179924us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.096782 (+1612525us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540027e554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:42.096990 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":412574464}
I20240507 08:06:42.101137 13399 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:42.151062 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.102780 13121 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240507 08:06:42.112809 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2157) took 1139 ms. Trace:
I20240507 08:06:42.110080 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1852) took 1979 ms. Trace:
W20240507 08:06:42.084393 13491 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.214s user 0.000s sys 0.000s
I20240507 08:06:42.113179 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.176898 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.085198 13327 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:42.108474 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.178351 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.178391 10232 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:42.179020 10185 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:42.179118 13530 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.180704 13530 consensus_queue.cc:1046] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:42.182166 13429 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.183096 13327 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.183722 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.184182 13429 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.185040 13415 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.186097 13280 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.186583 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.187388 10938 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:42.187741 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:42.188453 13332 consensus_meta.cc:287] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.756s user 0.004s sys 0.004s
I20240507 08:06:42.188725 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.064779 13326 raft_consensus.cc:686] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.119681 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.189817 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 514) took 1285 ms. Trace:
I20240507 08:06:42.189898 10191 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:42.190925 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.191460 10191 raft_consensus.cc:3037] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:42.190312 10267 rpcz_store.cc:276] 0507 08:06:40.904414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.069694 (+165280us) service_pool.cc:225] Handling call
0507 08:06:41.069985 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:41.070387 (+ 402us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:41.070415 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:41.083742 (+ 13327us) log.cc:842] Serialized 26 byte log entry
0507 08:06:41.369468 (+285726us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:41.369507 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:41.369576 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:42.189634 (+820058us) raft_consensus.cc:1671] finished
0507 08:06:42.189652 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.189765 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3648,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":8852,"spinlock_wait_cycles":1521408,"thread_start_us":278691,"threads_started":1,"wal-append.queue_time_us":659631}
I20240507 08:06:42.192773 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.193270 10267 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:42.063230 13367 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.122867 10304 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.122218 13523 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20240507 08:06:42.091389 13524 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.194716 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.146625 13506 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:42.102476 13505 log.cc:924] Time spent T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.318s user 0.001s sys 0.000s
I20240507 08:06:42.199304 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:42.201248 13526 raft_consensus.cc:3032] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:42.201709 13526 raft_consensus.cc:729] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.158596 10187 rpcz_store.cc:276] 0507 08:06:40.130957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.469023 (+338066us) service_pool.cc:225] Handling call
0507 08:06:40.482044 (+ 13021us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:40.504073 (+ 22029us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:40.504105 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:40.504143 (+ 38us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:40.504173 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.109881 (+1605708us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b5400260554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:42.109925 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.110032 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50237184}
I20240507 08:06:42.160318 13121 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:42.113384 13086 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.202354 13121 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:42.202627 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.203230 10187 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:42.203701 13086 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:42.161860 13276 consensus_meta.cc:287] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.501s user 0.006s sys 0.003s
I20240507 08:06:42.190908 13346 consensus_queue.cc:1035] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20240507 08:06:42.204723 13276 raft_consensus.cc:504] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.196724 13523 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:42.205996 13523 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:42.206655 10192 raft_consensus.cc:2379] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 8: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
W20240507 08:06:42.162741 13418 consensus_meta.cc:287] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.751s user 0.009s sys 0.000s
I20240507 08:06:42.212229 10191 raft_consensus.cc:2454] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:42.164712 10306 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:42.158253 10336 rpcz_store.cc:276] 0507 08:06:40.973172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.163003 (+189831us) service_pool.cc:225] Handling call
0507 08:06:41.163297 (+ 294us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:41.214601 (+ 51304us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:41.214633 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:41.231912 (+ 17279us) log.cc:842] Serialized 26 byte log entry
0507 08:06:41.489166 (+257254us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:41.675852 (+186686us) log.cc:842] Serialized 14 byte log entry
0507 08:06:41.925186 (+249334us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:41.925464 (+ 278us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:42.112590 (+187126us) raft_consensus.cc:1671] finished
0507 08:06:42.112611 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.112751 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137016,"prepare.queue_time_us":166422,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":10786,"raft.queue_time_us":413700,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":7034,"thread_start_us":564681,"threads_started":3,"wal-append.queue_time_us":389165}
I20240507 08:06:42.216950 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.217118 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.218170 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.219348 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.220283 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:42.221372 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 98647.5us
I20240507 08:06:42.221748 13322 raft_consensus.cc:2726] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:42.121747 13095 raft_consensus.cc:2781] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:42.222321 10304 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.222678 13095 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:42.223098 13095 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:42.223451 13367 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.223629 10304 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:42.225118 13415 raft_consensus.cc:2726] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240507 08:06:42.169806 13522 raft_consensus.cc:2726] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:42.216171 13276 leader_election.cc:288] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.225764 13415 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:42.226253 13415 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:42.189805 13326 consensus_queue.cc:237] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.226613 13415 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
W20240507 08:06:42.170799 13426 consensus_meta.cc:287] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.800s user 0.010s sys 0.000s
W20240507 08:06:42.171275 10190 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2231) took 2372 ms (client timeout 1639 ms). Trace:
W20240507 08:06:42.231660 10190 rpcz_store.cc:269] 0507 08:06:39.798799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:39.935023 (+136224us) service_pool.cc:225] Handling call
0507 08:06:42.171035 (+2236012us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b540005f354. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:42.171168 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3885628288}
I20240507 08:06:42.150022 13117 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240507 08:06:42.173857 13508 consensus_queue.cc:237] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.174715 13521 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:42.175666 13409 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:42.208420 13375 consensus_meta.cc:287] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.559s user 0.005s sys 0.005s
I20240507 08:06:42.192956 13469 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.188969 10185 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:42.181511 13530 raft_consensus.cc:2781] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20240507 08:06:42.198505 13510 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:42.246261 10155 leader_election.cc:302] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.204272 13524 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.207196 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.197863 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f47233c3be4fe99e097ade2aaec1e8" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.164599 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.207619 13141 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.227866 10187 raft_consensus.cc:2454] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:42.228044 10267 raft_consensus.cc:2454] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 7.
W20240507 08:06:42.180723 13336 consensus_meta.cc:287] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.723s user 0.004s sys 0.005s
I20240507 08:06:42.165411 13519 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:42.188649 13460 consensus_queue.cc:1035] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
W20240507 08:06:42.169605 13364 consensus_meta.cc:287] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.667s user 0.009s sys 0.000s
I20240507 08:06:42.202471 13526 consensus_queue.cc:260] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.230312 13418 raft_consensus.cc:2726] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081) with higher term. Message: Invalid argument: T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:42.229382 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.188874 13332 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:42.230683 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 66075.2us
I20240507 08:06:42.267596 10306 leader_election.cc:302] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.233821 10232 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:42.176003 10184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.235225 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.231038 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.242843 10908 leader_election.cc:302] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540; no voters:
I20240507 08:06:42.243606 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.243638 10156 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:42.247089 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 131124us
I20240507 08:06:42.228942 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03d2e621edea41ae97e79e36e4dc0365" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.259691 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.246392 13121 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.262673 13526 raft_consensus.cc:3037] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:42.263598 10186 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240507 08:06:42.270097 10191 raft_consensus.cc:2454] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:42.264492 10336 raft_consensus.cc:2354] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:42.270045 10190 raft_consensus.cc:2379] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 7: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
I20240507 08:06:42.271855 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.276639 10232 leader_election.cc:302] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:42.272902 10231 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.273947 10184 raft_consensus.cc:1226] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20240507 08:06:42.158664 13403 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:42.255353 13458 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540: Append to log took a long time: real 1.490s user 0.001s sys 0.000s
I20240507 08:06:42.245832 13530 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:42.215700 13506 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.256397 13455 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20240507 08:06:42.280910 13169 raft_consensus.cc:2781] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:42.261590 13336 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.283035 13141 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.286273 13524 raft_consensus.cc:2781] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:42.286401 13095 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.286836 13332 leader_election.cc:288] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.255681 13276 raft_consensus.cc:2726] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.287150 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.306026 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.306337 13095 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.306701 10336 raft_consensus.cc:2379] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 7: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:42.307431 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.308158 10186 raft_consensus.cc:1226] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 4. Current term is 5. Ops: []
I20240507 08:06:42.308375 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.288007 10232 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:42.310878 10188 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:42.288337 13469 leader_election.cc:288] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.288084 13521 consensus_queue.cc:1046] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:42.260589 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:42.288694 13374 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.292770 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.322970 10260 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:42.323064 10263 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:42.295675 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.256733 13527 raft_consensus.cc:2726] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20240507 08:06:42.302623 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 59111.6us
I20240507 08:06:42.331821 10156 leader_election.cc:302] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.332355 13422 raft_consensus.cc:3032] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:42.332820 13422 raft_consensus.cc:729] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.287580 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.260968 13364 raft_consensus.cc:504] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.291524 13490 raft_consensus.cc:2781] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:42.242430 13375 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:42.322255 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 54765.3us
I20240507 08:06:42.337209 10337 raft_consensus.cc:2454] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
W20240507 08:06:42.342304 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 46724.7us
I20240507 08:06:42.298275 13530 raft_consensus.cc:3037] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:42.313516 10906 leader_election.cc:302] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:42.333679 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 45772.2us
I20240507 08:06:42.264266 13519 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.314487 13418 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.338078 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.339100 10156 leader_election.cc:398] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7.
I20240507 08:06:42.296861 13117 consensus_queue.cc:1046] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20240507 08:06:42.343593 10231 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.377825 10231 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:42.378865 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 130286us
I20240507 08:06:42.378876 10231 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:42.347209 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.331127 13524 raft_consensus.cc:686] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.330976 13374 raft_consensus.cc:2726] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240507 08:06:42.374954 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.382937 10191 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:42.346608 13326 raft_consensus.cc:2726] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:42.375677 13529 raft_consensus.cc:2726] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.376497 13403 leader_election.cc:288] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.381376 10155 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.388885 10263 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:42.336838 13169 raft_consensus.cc:686] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.352798 13537 raft_consensus.cc:2781] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240507 08:06:42.384919 10306 leader_election.cc:302] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:42.383873 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.391060 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.377709 13332 consensus_queue.cc:1046] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20240507 08:06:42.395044 10155 leader_election.cc:302] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.397032 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.381644 13121 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.398070 10191 raft_consensus.cc:2354] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:42.380262 13421 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:42.382834 13117 raft_consensus.cc:2726] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.398198 10155 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.398573 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.403753 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.407866 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.408416 10191 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:42.409315 10232 leader_election.cc:398] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:42.409644 10232 leader_election.cc:401] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20240507 08:06:42.410593 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.411222 10263 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:42.410012 13421 raft_consensus.cc:504] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.444610 10185 raft_consensus.cc:1226] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [term 10 LEADER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:42.399506 13403 raft_consensus.cc:2781] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:42.412052 13447 raft_consensus.cc:2781] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:42.411806 13524 consensus_queue.cc:237] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.391897 13548 raft_consensus.cc:2726] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.416432 13506 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.416836 13409 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.418002 13532 raft_consensus.cc:2726] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20240507 08:06:42.418661 13364 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.419667 13519 leader_election.cc:288] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.396871 13510 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.421613 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.423727 13399 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.425956 13415 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.472980 13336 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.432471 10304 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.444306 13521 raft_consensus.cc:3032] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Stepping down as leader of term 4
W20240507 08:06:42.446379 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 67603.7us
I20240507 08:06:42.411556 13332 raft_consensus.cc:2741] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20240507 08:06:42.461182 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.451236 13522 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:42.462679 13169 consensus_queue.cc:237] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.403678 13537 raft_consensus.cc:482] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:42.483670 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 51292.8us
I20240507 08:06:42.484140 13519 raft_consensus.cc:2726] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:42.459769 13117 raft_consensus.cc:3037] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:42.437209 10155 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:42.494722 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 57679.8us
I20240507 08:06:42.486161 10231 leader_election.cc:302] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.464109 13403 raft_consensus.cc:686] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.496673 10231 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:42.505232 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.504182 10231 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:42.485729 13447 raft_consensus.cc:686] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:42.487412 13521 raft_consensus.cc:729] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.518692 10231 leader_election.cc:302] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.486563 13490 raft_consensus.cc:686] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [term 8 LEADER]: Becoming Leader. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:42.520929 10263 raft_consensus.cc:2454] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:42.517434 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.517871 13414 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:42.513895 13121 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:42.532567 13121 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:42.533021 13121 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:42.532788 13521 consensus_queue.cc:260] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.534026 13521 raft_consensus.cc:3037] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Advancing to term 5
I20240507 08:06:42.519276 10191 raft_consensus.cc:2454] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 4.
I20240507 08:06:42.505899 13492 raft_consensus.cc:2781] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:42.505048 13522 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.505445 13407 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.522787 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.543108 10187 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:42.510411 13336 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:42.523561 13490 consensus_queue.cc:237] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.524422 13529 raft_consensus.cc:2726] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.525172 13367 raft_consensus.cc:2726] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.546433 13403 consensus_queue.cc:237] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.506176 13549 raft_consensus.cc:2726] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:42.530280 10938 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:42.524693 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.554996 10186 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:42.522329 13444 log.cc:924] Time spent T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 2.044s user 0.000s sys 0.005s
I20240507 08:06:42.535831 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbe630c1736458898143fb230a9ffb8" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.493479 13375 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.563462 13367 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:42.537217 10306 leader_election.cc:302] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:42.514379 13421 leader_election.cc:288] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.566572 10191 raft_consensus.cc:1226] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:42.567734 10191 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:42.564042 13367 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.494581 13537 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Advancing to term 11
W20240507 08:06:42.548602 13462 log.cc:924] Time spent T e9adf1535c974b2abf910f3c1c537627 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 1.689s user 0.000s sys 0.003s
I20240507 08:06:42.548981 13415 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.549913 13336 raft_consensus.cc:686] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.536290 13414 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:42.551841 13407 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:42.545668 13492 raft_consensus.cc:686] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:42.561640 13406 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:42.558626 13522 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.577538 10306 leader_election.cc:302] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:42.576297 13421 raft_consensus.cc:2781] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 FOLLOWER]: Leader election won for term 4
W20240507 08:06:42.587832 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 66872.6us
I20240507 08:06:42.586719 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.597203 10191 raft_consensus.cc:1261] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240507 08:06:42.520493 10231 leader_election.cc:302] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
W20240507 08:06:42.598608 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 78209.2us
I20240507 08:06:42.589509 13399 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.605192 10186 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:42.600433 13522 raft_consensus.cc:2726] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240507 08:06:42.606742 13336 consensus_queue.cc:237] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.610953 10232 leader_election.cc:302] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:42.585016 13320 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20240507 08:06:42.599850 13415 raft_consensus.cc:2726] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:42.622886 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 436) took 1184 ms. Trace:
I20240507 08:06:42.624886 10334 rpcz_store.cc:276] 0507 08:06:41.438372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.513047 (+ 74675us) service_pool.cc:225] Handling call
0507 08:06:41.583146 (+ 70099us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:41.588387 (+ 5241us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:41.588417 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:41.588456 (+ 39us) raft_consensus.cc:1631] Marking committed up to 9
0507 08:06:41.588485 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.622739 (+1034254us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b54003c4454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:42.622770 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.622846 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11554048}
I20240507 08:06:42.626619 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 1987) took 2330 ms. Trace:
I20240507 08:06:42.602998 13421 raft_consensus.cc:686] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.627002 10265 rpcz_store.cc:276] 0507 08:06:40.295953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.825904 (+529951us) service_pool.cc:225] Handling call
0507 08:06:42.097382 (+1271478us) raft_consensus.cc:1457] Updating replica for 2 ops
0507 08:06:42.101622 (+ 4240us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:42.101653 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0507 08:06:42.102217 (+ 564us) log.cc:842] Serialized 52 byte log entry
0507 08:06:42.259207 (+156990us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:42.259242 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.259299 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:42.329747 (+ 70448us) raft_consensus.cc:1671] finished
0507 08:06:42.329766 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.622273 (+292507us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540027e550. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:42.626560 (+ 4287us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":30037,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":16617,"spinlock_wait_cycles":2923961728,"thread_start_us":156629,"threads_started":1,"wal-append.queue_time_us":204851}
I20240507 08:06:42.628865 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.629390 10265 raft_consensus.cc:3037] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:42.640174 10186 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:42.593693 13407 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.646670 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.595824 13406 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.647970 10187 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:42.620985 13403 raft_consensus.cc:2781] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:42.650033 10306 leader_election.cc:302] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
I20240507 08:06:42.610692 13455 consensus_queue.cc:1046] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20240507 08:06:42.648047 10265 raft_consensus.cc:2454] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 6.
I20240507 08:06:42.652690 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.652770 10304 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:42.653647 10265 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:42.643465 13332 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:42.657061 13332 raft_consensus.cc:504] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.657675 13374 raft_consensus.cc:2781] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:42.658442 13455 raft_consensus.cc:3032] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:42.658838 13455 raft_consensus.cc:729] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.659180 13332 leader_election.cc:288] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.659641 13455 consensus_queue.cc:260] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:42.660900 13406 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.661792 13374 raft_consensus.cc:686] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.662547 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.662775 13374 consensus_queue.cc:237] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.549943 13447 consensus_queue.cc:237] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.665676 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1865) took 1255 ms. Trace:
I20240507 08:06:42.667599 10188 rpcz_store.cc:276] 0507 08:06:41.410276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.429970 (+ 19694us) service_pool.cc:225] Handling call
0507 08:06:42.665508 (+1235538us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400046854. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:42.665622 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1945906560}
I20240507 08:06:42.672259 10186 raft_consensus.cc:1261] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:42.673207 10186 raft_consensus.cc:1226] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:42.678893 13198 consensus_queue.cc:1035] T 3490f6d2f719474bb4f72a5647a9a726 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:42.679214 13117 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:42.679121 13415 consensus_queue.cc:1046] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240507 08:06:42.680331 13198 raft_consensus.cc:3032] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:42.680675 13198 raft_consensus.cc:729] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.681183 13367 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.647583 13513 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:42.664875 13320 consensus_queue.cc:1046] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20240507 08:06:42.681931 13513 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.682657 13320 raft_consensus.cc:3032] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Stepping down as leader of term 10
I20240507 08:06:42.683017 13320 raft_consensus.cc:729] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.684195 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2001) took 1097 ms. Trace:
I20240507 08:06:42.684155 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.684724 10266 rpcz_store.cc:276] 0507 08:06:41.587217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.587342 (+ 125us) service_pool.cc:225] Handling call
0507 08:06:41.587640 (+ 298us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:42.683672 (+1096032us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b540010aa20. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90dbb328 00007fcf90e2723d 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc
0507 08:06:42.683744 (+ 72us) raft_consensus.cc:1520] Early marking committed up to index 6
0507 08:06:42.683783 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:42.683840 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:42.683900 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.683985 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:42.684127 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":688,"raft.queue_time_us":22618,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":228755840}
I20240507 08:06:42.687435 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.688992 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.585351 13095 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20240507 08:06:42.655689 13364 raft_consensus.cc:2781] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:42.691730 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd78797b4574a049447c358499ad846" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.696230 10334 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:42.692783 10266 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:42.698113 10304 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:42.660925 13455 raft_consensus.cc:3037] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:42.609889 13527 consensus_queue.cc:1035] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20240507 08:06:42.663249 13422 consensus_queue.cc:260] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.699410 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.702243 10265 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 11.
I20240507 08:06:42.711498 10186 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:42.712214 10155 leader_election.cc:302] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.714361 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.714448 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.716924 10265 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 7.
W20240507 08:06:42.719740 13523 consensus_meta.cc:287] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.513s user 0.005s sys 0.004s
I20240507 08:06:42.719437 13332 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:42.743945 10334 raft_consensus.cc:2454] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 6.
I20240507 08:06:42.745316 10156 leader_election.cc:302] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.752128 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324b71380c6c47bbb335661d299baea0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.725800 13364 raft_consensus.cc:686] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.664264 13492 consensus_queue.cc:237] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.768018 10334 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:42.757359 13530 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.753998 13319 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:42.765141 10333 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1720) took 1779 ms (client timeout 2233 ms). Trace:
W20240507 08:06:42.766112 10258 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 1992) took 2175 ms (client timeout 1841 ms). Trace:
I20240507 08:06:42.681383 13198 consensus_queue.cc:260] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.735615 13332 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:42.729938 13320 consensus_queue.cc:260] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:42.783159 10258 rpcz_store.cc:269] 0507 08:06:40.590558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.782536 (+191978us) service_pool.cc:225] Handling call
0507 08:06:42.765897 (+1983361us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b5400070dd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:42.766036 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":920182656}
I20240507 08:06:42.730533 13421 consensus_queue.cc:237] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.875383 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b89492a3abf415da273cc9a5b1227bf" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:42.782794 10333 rpcz_store.cc:269] 0507 08:06:40.985884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.053285 (+ 67401us) service_pool.cc:225] Handling call
0507 08:06:42.764933 (+1711648us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b54003c3f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:42.765073 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13332736}
I20240507 08:06:42.736011 13523 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:42.885819 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 173796us
I20240507 08:06:42.886942 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.754575 13506 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:42.825830 10937 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:42.888959 13487 raft_consensus.cc:2781] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Leader election won for term 11
I20240507 08:06:42.893016 13487 raft_consensus.cc:686] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [term 11 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:42.833143 13484 log.cc:924] Time spent T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 1.223s user 0.000s sys 0.005s
I20240507 08:06:42.890122 10155 leader_election.cc:302] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.881402 13336 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:42.891556 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
W20240507 08:06:42.886003 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 140873us
W20240507 08:06:42.897302 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 70313.8us
I20240507 08:06:42.904104 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.905071 10934 raft_consensus.cc:2379] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 6: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:42.903421 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:42.897730 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
W20240507 08:06:42.907198 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 261494us
I20240507 08:06:42.897164 13320 raft_consensus.cc:3037] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:42.782936 13513 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.900220 10264 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:42.890547 13441 raft_consensus.cc:2781] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election won for term 6
I20240507 08:06:42.888375 13332 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:42.827407 13461 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:42.885109 13407 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.883921 13403 raft_consensus.cc:686] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:42.909569 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.949432 10258 raft_consensus.cc:2379] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 10: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:42.898186 10334 raft_consensus.cc:2454] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 11.
I20240507 08:06:42.910713 10906 leader_election.cc:302] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:42.910704 13117 consensus_queue.cc:1046] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:42.908200 10333 raft_consensus.cc:2354] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:42.898855 13375 raft_consensus.cc:2726] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240507 08:06:42.894218 13487 consensus_queue.cc:237] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.901708 13198 raft_consensus.cc:3037] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
W20240507 08:06:42.933319 13526 consensus_meta.cc:287] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.642s user 0.002s sys 0.006s
I20240507 08:06:42.903877 13336 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.954265 13095 raft_consensus.cc:2726] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:42.960311 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.961192 13336 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.961422 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:42.935073 13523 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:42.935209 13553 log.cc:924] Time spent T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.206s user 0.000s sys 0.001s
I20240507 08:06:42.806687 13319 raft_consensus.cc:504] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.950260 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:42.964718 10334 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:42.911324 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.954885 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:42.955544 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:42.808368 13364 consensus_queue.cc:237] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.956521 10231 leader_election.cc:302] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:42.940521 13121 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.947914 13461 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.961876 13415 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:42.966650 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:42.966773 10232 leader_election.cc:302] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:42.954360 13326 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:42.896653 13506 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:42.954792 13441 raft_consensus.cc:686] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Leader. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:42.969130 10156 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:42.955497 13530 leader_election.cc:288] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:42.966027 10258 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:42.989719 10266 raft_consensus.cc:3037] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:42.973647 10187 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20240507 08:06:42.993973 13461 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:42.996244 13469 raft_consensus.cc:3032] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Stepping down as leader of term 9
I20240507 08:06:42.996613 13469 raft_consensus.cc:729] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:42.997555 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:42.983002 13415 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.002367 13415 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:42.973750 10186 raft_consensus.cc:2454] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:42.973989 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cdcc48b264f4a9eb3eabce20e5bbd6c" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.003162 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.005630 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.006696 10267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed7ce583c04c48ebb2e80d2e0e872215" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.007216 10267 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:42.988922 13336 raft_consensus.cc:2726] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240507 08:06:42.997505 13469 consensus_queue.cc:260] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.009007 13469 raft_consensus.cc:3037] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:42.977288 10259 raft_consensus.cc:2454] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d6656a616b747d393ff3de5328d2540 in term 8.
I20240507 08:06:43.008826 13441 consensus_queue.cc:237] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:42.983440 10334 raft_consensus.cc:2454] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:42.982702 13526 raft_consensus.cc:2781] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Leader election won for term 11
I20240507 08:06:42.976146 10231 leader_election.cc:398] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 election: Vote denied by peer 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047) with higher term. Message: Invalid argument: T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7.
I20240507 08:06:42.956081 13562 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:43.090366 10261 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
W20240507 08:06:43.053691 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 84697.5us
I20240507 08:06:43.010829 10934 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:43.051508 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:43.023048 10262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad8dd18470a542d3bec18f13b451878d" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:43.083549 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1878) took 1060 ms. Trace:
I20240507 08:06:43.071197 10337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:43.009443 10189 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55820 (request call id 2244) took 2221 ms (client timeout 2105 ms). Trace:
I20240507 08:06:43.033715 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a774a8075d4c689b047bb943c78c50" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.084381 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2169) took 1015 ms. Trace:
I20240507 08:06:43.053197 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1861) took 1666 ms. Trace:
W20240507 08:06:43.153398 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 177504us
I20240507 08:06:43.148643 10186 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:43.056764 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 431) took 2040 ms. Trace:
I20240507 08:06:43.067088 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da12b367b046b7bcfb08aaa6e1c97b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.161022 10192 rpcz_store.cc:276] 0507 08:06:41.386774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.437879 (+ 51105us) service_pool.cc:225] Handling call
0507 08:06:43.053012 (+1615133us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54001beed4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.053134 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1718418816}
I20240507 08:06:43.162221 10335 rpcz_store.cc:276] 0507 08:06:41.016356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.221946 (+205590us) service_pool.cc:225] Handling call
0507 08:06:42.765073 (+1543127us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:42.765260 (+ 187us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:42.765289 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:42.765333 (+ 44us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:42.765365 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:42.765422 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.052740 (+287318us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54003c3f50. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.056700 (+ 3960us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3549788416}
I20240507 08:06:43.024164 10267 raft_consensus.cc:2454] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:43.077762 10907 leader_election.cc:302] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:43.164130 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:43.164546 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 86955.8us
I20240507 08:06:43.031414 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 434) took 1708 ms. Trace:
I20240507 08:06:43.168888 10306 leader_election.cc:302] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:43.168629 10339 rpcz_store.cc:276] 0507 08:06:41.323191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.323375 (+ 184us) service_pool.cc:225] Handling call
0507 08:06:41.323744 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:41.382209 (+ 58465us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:41.382241 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:41.382279 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:41.382314 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.031218 (+1648904us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b540013f454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:43.031261 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.031363 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134035712}
I20240507 08:06:43.078423 10266 raft_consensus.cc:2454] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:43.045518 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2261) took 1101 ms. Trace:
I20240507 08:06:43.170472 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a210876eb5b4a7ab6548839daccfe84" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.042989 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2000) took 1456 ms. Trace:
I20240507 08:06:43.165624 10231 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
W20240507 08:06:43.025913 10340 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1719) took 2052 ms (client timeout 2511 ms). Trace:
I20240507 08:06:43.024374 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2159) took 2037 ms. Trace:
I20240507 08:06:43.165553 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.173166 10186 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 in current term 6: Already voted for candidate 59382e6647c549d89d83c70233c0667e in this term.
W20240507 08:06:43.160071 10189 rpcz_store.cc:269] 0507 08:06:40.788195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.827629 (+ 39434us) service_pool.cc:225] Handling call
0507 08:06:43.009256 (+2181627us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b540005f854. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.009376 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21127424}
I20240507 08:06:43.058923 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 430) took 2043 ms. Trace:
W20240507 08:06:43.171726 10340 rpcz_store.cc:269] 0507 08:06:40.973568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.075428 (+101860us) service_pool.cc:225] Handling call
0507 08:06:43.025745 (+1950317us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b5400069354. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.025858 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2151720320}
I20240507 08:06:43.165912 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 2040) took 1071 ms. Trace:
I20240507 08:06:43.050091 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.068032 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfb70b9953fd486ca5f88b5999dd26a0" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.170276 10185 rpcz_store.cc:276] 0507 08:06:41.944510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:42.297372 (+352862us) service_pool.cc:225] Handling call
0507 08:06:42.319373 (+ 22001us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:42.445110 (+125737us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.045218 (+600108us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400046d54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:43.045454 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":254708608}
I20240507 08:06:43.172091 10341 rpcz_store.cc:276] 0507 08:06:40.986980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.215612 (+228632us) service_pool.cc:225] Handling call
0507 08:06:41.219852 (+ 4240us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:41.220396 (+ 544us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.024147 (+1803751us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b5400334850. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.024304 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9001856}
I20240507 08:06:43.174172 10338 rpcz_store.cc:276] 0507 08:06:41.015900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.219303 (+203403us) service_pool.cc:225] Handling call
0507 08:06:41.219549 (+ 246us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:41.299035 (+ 79486us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:41.299082 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:41.299137 (+ 55us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:41.299180 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.058700 (+1759520us) spinlock_profiling.cc:243] Waited 79.3 ms on lock 0x7b540013e554. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:43.058752 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.058870 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182345984}
I20240507 08:06:43.159466 10184 rpcz_store.cc:276] 0507 08:06:42.023591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:42.417287 (+393696us) service_pool.cc:225] Handling call
0507 08:06:42.418963 (+ 1676us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:42.469621 (+ 50658us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:42.469652 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:42.469690 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:42.469723 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.083281 (+613558us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b54001be4d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:43.083339 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.083483 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116013440}
I20240507 08:06:43.176481 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.177217 10338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d914d033cf48c389262dd8b7c7fbce" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.178220 10261 rpcz_store.cc:276] 0507 08:06:42.094915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.089801 (+994886us) service_pool.cc:225] Handling call
0507 08:06:43.090151 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.165699 (+ 75548us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.165866 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:43.160697 10336 rpcz_store.cc:276] 0507 08:06:42.068916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:42.341956 (+273040us) service_pool.cc:225] Handling call
0507 08:06:43.071140 (+729184us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.071430 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.071459 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:43.071497 (+ 38us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:43.071524 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.071576 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.084197 (+ 12621us) spinlock_profiling.cc:243] Waited 729 ms on lock 0x7b540013ea50. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.084317 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":882,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":86,"spinlock_wait_cycles":1677157760}
I20240507 08:06:43.183768 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.171411 10260 rpcz_store.cc:276] 0507 08:06:41.586322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.586447 (+ 125us) service_pool.cc:225] Handling call
0507 08:06:41.627483 (+ 41036us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:42.323564 (+696081us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.042802 (+719238us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b54002bfa54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:43.042938 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1599700224}
I20240507 08:06:43.184437 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a123878203f45319def3c4246b9fe30" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.172922 10258 raft_consensus.cc:2454] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:43.185377 10306 leader_election.cc:302] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.188953 10267 raft_consensus.cc:1261] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:43.189783 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 2042) took 1089 ms. Trace:
I20240507 08:06:43.190107 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 438) took 1112 ms. Trace:
I20240507 08:06:43.190097 10267 rpcz_store.cc:276] 0507 08:06:42.100627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.184456 (+1083829us) service_pool.cc:225] Handling call
0507 08:06:43.184722 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.189543 (+ 4821us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.189735 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:43.190797 10340 rpcz_store.cc:276] 0507 08:06:42.077288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.189361 (+1112073us) service_pool.cc:225] Handling call
0507 08:06:43.189635 (+ 274us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.189801 (+ 166us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.189833 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:43.189874 (+ 41us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:43.189919 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.189968 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.190060 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:43.189069 10259 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20240507 08:06:42.978708 13521 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:43.190045 10337 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:43.174535 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.204437 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 439) took 1102 ms. Trace:
I20240507 08:06:43.204757 10337 rpcz_store.cc:276] 0507 08:06:42.101921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.189567 (+1087646us) service_pool.cc:225] Handling call
0507 08:06:43.189886 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.204207 (+ 14321us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.204387 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:43.126210 13559 log.cc:924] Time spent T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
W20240507 08:06:43.010214 13558 log.cc:924] Time spent T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240507 08:06:43.200910 10341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8933e79c6c78466498a6d17d93ce7cff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.202945 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.044757 13426 raft_consensus.cc:3032] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:43.206005 10258 raft_consensus.cc:1226] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 7. Current term is 8. Ops: []
W20240507 08:06:43.073837 13504 log.cc:924] Time spent T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 1.206s user 0.004s sys 0.000s
I20240507 08:06:43.206507 10340 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:43.207113 10231 leader_election.cc:302] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:43.221899 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.222368 10231 leader_election.cc:302] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:43.222781 10189 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
W20240507 08:06:43.152768 13475 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.699s user 0.006s sys 0.012s
I20240507 08:06:43.208384 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.055440 13157 raft_consensus.cc:3037] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:43.093945 13523 raft_consensus.cc:3032] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:43.054258 13443 raft_consensus.cc:3032] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Stepping down as leader of term 5
W20240507 08:06:43.203399 10262 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 2024) took 2659 ms (client timeout 1831 ms). Trace:
I20240507 08:06:43.183409 13533 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:43.022286 13422 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:43.053079 13346 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:43.213663 10341 raft_consensus.cc:2454] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:43.215319 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.060827 13375 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.166s
I20240507 08:06:43.042526 13361 leader_election.cc:288] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.036862 13205 raft_consensus.cc:482] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting leader election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:43.087306 13319 leader_election.cc:288] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:43.223907 10339 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:43.074121 13537 raft_consensus.cc:504] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.276185 13513 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:43.276944 13513 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.278023 13537 leader_election.cc:288] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.278771 10265 raft_consensus.cc:2379] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 11: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:43.279949 10267 raft_consensus.cc:1226] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:43.215587 13521 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.288134 10089 catalog_manager.cc:5484] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to 59382e6647c549d89d83c70233c0667e (127.8.213.129). New cstate: current_term: 6 leader_uuid: "59382e6647c549d89d83c70233c0667e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:43.265087 13205 raft_consensus.cc:3037] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:43.070266 13121 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.225354 10231 leader_election.cc:302] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:43.292558 13121 raft_consensus.cc:2726] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:43.088320 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9adf1535c974b2abf910f3c1c537627" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:43.166812 13420 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:43.298173 13420 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.167040 13564 raft_consensus.cc:482] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:43.093458 13519 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:43.227926 10260 raft_consensus.cc:1226] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:43.227880 10340 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:43.302888 10155 leader_election.cc:302] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:43.226151 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:43.304514 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 78486.5us
I20240507 08:06:43.188644 13548 raft_consensus.cc:2781] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election won for term 10
W20240507 08:06:43.236948 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 270391us
W20240507 08:06:43.238792 10262 rpcz_store.cc:269] 0507 08:06:40.523709 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:40.752828 (+229119us) service_pool.cc:225] Handling call
0507 08:06:43.022858 (+2270030us) spinlock_profiling.cc:243] Waited 7.88 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:06:43.183351 (+160493us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18132352}
I20240507 08:06:43.050034 13147 raft_consensus.cc:3032] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Stepping down as leader of term 5
I20240507 08:06:43.202515 13455 raft_consensus.cc:2726] T 84da12b367b046b7bcfb08aaa6e1c97b P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:43.188136 13526 raft_consensus.cc:686] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:43.220983 13426 raft_consensus.cc:729] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:43.247536 13466 consensus_queue.cc:1046] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20240507 08:06:43.226665 13475 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.776s user 0.006s sys 0.012s
I20240507 08:06:43.233950 13141 raft_consensus.cc:2726] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240507 08:06:43.248245 10185 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:43.215963 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.253885 10335 raft_consensus.cc:1226] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:43.322762 10191 raft_consensus.cc:1226] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:42.986465 13326 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.256765 10263 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:43.206571 13415 consensus_queue.cc:1046] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20240507 08:06:43.234200 13530 raft_consensus.cc:2726] T 50d914d033cf48c389262dd8b7c7fbce P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20240507 08:06:43.234187 13337 raft_consensus.cc:2726] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20240507 08:06:43.319092 13527 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:43.037101 13086 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:43.238718 13443 raft_consensus.cc:729] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:43.085866 13414 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.275776 10339 raft_consensus.cc:2454] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:43.303227 13418 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20240507 08:06:43.279492 13513 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:43.377525 10339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.225757 10336 raft_consensus.cc:1261] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:43.379539 10336 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2173) took 1283 ms. Trace:
I20240507 08:06:43.348193 13420 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.379940 10336 rpcz_store.cc:276] 0507 08:06:42.096003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.195925 (+1099922us) service_pool.cc:225] Handling call
0507 08:06:43.225312 (+ 29387us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.379231 (+153919us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.379461 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:43.313601 13564 raft_consensus.cc:504] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.293272 10089 catalog_manager.cc:5484] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:43.382478 10089 catalog_manager.cc:5484] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 7, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 7 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:43.383911 10089 catalog_manager.cc:5484] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:43.234501 13523 raft_consensus.cc:729] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:43.300823 10307 leader_election.cc:302] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:43.303047 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2013) took 1004 ms. Trace:
I20240507 08:06:43.371201 13086 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.385648 10260 rpcz_store.cc:276] 0507 08:06:42.298626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.191809 (+893183us) service_pool.cc:225] Handling call
0507 08:06:43.227768 (+ 35959us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.302824 (+ 75056us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.302998 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:43.386659 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:43.312008 10155 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.285259 13521 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:43.314509 10192 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:43.313045 10304 leader_election.cc:302] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:43.318003 10934 raft_consensus.cc:1226] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:43.315953 10232 leader_election.cc:302] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:43.362767 13527 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:43.124167 13403 consensus_queue.cc:237] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.329332 10087 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59936 (request call id 216) took 6561 ms. Trace:
I20240507 08:06:43.315825 13121 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:43.183298 13562 raft_consensus.cc:504] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.313958 13412 raft_consensus.cc:2781] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:43.348768 13487 consensus_queue.cc:1046] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:43.188128 13423 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20240507 08:06:43.313768 13147 raft_consensus.cc:729] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.360531 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.315734 13461 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:43.361274 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:43.360602 10191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "292ec14f14384ba1bc9ee81980e1eb2a" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:43.442816 10191 raft_consensus.cc:2354] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9.
I20240507 08:06:43.326552 13475 raft_consensus.cc:686] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.363539 10306 leader_election.cc:302] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:43.366501 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.367256 10263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:43.450606 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 87218.7us
I20240507 08:06:43.320139 13548 raft_consensus.cc:686] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
W20240507 08:06:43.292677 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 67448.4us
I20240507 08:06:43.370765 10340 raft_consensus.cc:1226] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:43.429620 13566 raft_consensus.cc:2781] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:43.299839 10183 raft_consensus.cc:2396] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20240507 08:06:43.385478 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 84822.2us
I20240507 08:06:43.313524 13519 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.459790 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:43.392705 10192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85b6fa5360054f32af996ec68d175584" candidate_uuid: "5d6656a616b747d393ff3de5328d2540" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.462507 10192 raft_consensus.cc:1261] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240507 08:06:43.300081 13375 consensus_queue.cc:1225] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer 5d6656a616b747d393ff3de5328d2540 log is divergent from this leader: its last log entry 7.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20240507 08:06:43.393054 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 77240.3us
I20240507 08:06:43.395951 10339 raft_consensus.cc:1226] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [term 4 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 3. Current term is 4. Ops: []
I20240507 08:06:43.389299 10089 catalog_manager.cc:5484] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 8 to 9, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 9 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:43.381515 10336 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:43.377249 13466 consensus_queue.cc:1046] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20240507 08:06:43.414655 13377 raft_consensus.cc:3032] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:43.414948 13414 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.415529 13326 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:43.416097 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 104250us
I20240507 08:06:43.416745 10188 raft_consensus.cc:1226] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 10. Current term is 11. Ops: []
I20240507 08:06:43.417171 13157 raft_consensus.cc:2726] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:43.429505 13537 consensus_queue.cc:1046] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20240507 08:06:43.424576 10087 rpcz_store.cc:276] 0507 08:06:36.756269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:36.756451 (+ 182us) service_pool.cc:225] Handling call
0507 08:06:43.317333 (+6560882us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:06:40.747880 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:06:40.841833 (+ 93953us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:06:40.841869 (+ 36us) write_op.cc:435] Acquired schema lock
0507 08:06:40.841884 (+ 15us) tablet.cc:660] Decoding operations
0507 08:06:40.843247 (+ 1363us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:06:40.843315 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0507 08:06:40.843344 (+ 29us) tablet.cc:683] Acquiring locks for 15 operations
0507 08:06:40.844341 (+ 997us) tablet.cc:699] Row locks acquired
0507 08:06:40.844359 (+ 18us) write_op.cc:260] PREPARE: finished
0507 08:06:40.844491 (+ 132us) write_op.cc:270] Start()
0507 08:06:40.844597 (+ 106us) write_op.cc:276] Timestamp: P: 1715069200844442 usec, L: 0
0507 08:06:40.844619 (+ 22us) op_driver.cc:331] REPLICATION: starting
0507 08:06:40.845258 (+ 639us) log.cc:842] Serialized 5964 byte log entry
0507 08:06:41.864546 (+1019288us) op_driver.cc:444] REPLICATION: finished
0507 08:06:42.224532 (+359986us) write_op.cc:301] APPLY: starting
0507 08:06:42.224613 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0507 08:06:42.224880 (+ 267us) tablet.cc:1368] finished BulkCheckPresence
0507 08:06:42.224995 (+ 115us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:06:42.226540 (+ 1545us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:06:42.226803 (+ 263us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0507 08:06:42.226848 (+ 45us) write_op.cc:312] APPLY: finished
0507 08:06:42.227238 (+ 390us) log.cc:842] Serialized 107 byte log entry
0507 08:06:42.240884 (+ 13646us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:42.256976 (+ 16092us) write_op.cc:454] Released schema lock
0507 08:06:42.257037 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0507 08:06:42.257317 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:06:42.257345 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":972410,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":347991,"apply.run_cpu_time_us":3967,"apply.run_wall_time_us":33289,"num_ops":15,"prepare.queue_time_us":548224,"prepare.run_cpu_time_us":23547,"prepare.run_wall_time_us":339945,"raft.queue_time_us":535793,"raft.run_cpu_time_us":40384,"raft.run_wall_time_us":244987,"replication_time_us":970761,"thread_start_us":517360,"threads_started":4,"wal-append.queue_time_us":272597,"wal-append.run_cpu_time_us":13735,"wal-append.run_wall_time_us":2121235}]]}
I20240507 08:06:43.370536 13115 raft_consensus.cc:2781] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20240507 08:06:43.387352 13086 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:43.432714 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "765054c26d4a4181b4fb31891d033bb1" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.435000 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0778bf2d68148c495412379f7818b28" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:43.361541 13460 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.061s
I20240507 08:06:43.440294 10187 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:43.430315 13526 consensus_queue.cc:237] T 324b71380c6c47bbb335661d299baea0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.445446 13332 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.433434 13310 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20240507 08:06:43.428542 13527 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:43.417935 13274 raft_consensus.cc:2726] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:43.479916 13566 raft_consensus.cc:686] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.255051 13533 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.564123 10334 raft_consensus.cc:2379] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 9: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:43.464021 10908 leader_election.cc:302] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5d6656a616b747d393ff3de5328d2540; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:43.468771 10089 catalog_manager.cc:5484] T f0778bf2d68148c495412379f7818b28 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 7, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 7 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:43.437314 13412 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:43.474238 10934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:43.385483 13523 consensus_queue.cc:260] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:43.419713 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 106815us
I20240507 08:06:43.477339 13455 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.073s
I20240507 08:06:43.459352 13532 raft_consensus.cc:482] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:43.520834 10186 raft_consensus.cc:1261] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20240507 08:06:43.521912 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.460508 13374 raft_consensus.cc:2726] T beb5387c55024337b72bb8aa1ba09a7b P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:43.476591 13442 consensus_queue.cc:1046] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240507 08:06:43.528733 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9cd8b1faef46a9bbbee733353fd123" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:43.596869 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:43.598003 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:43.528929 10155 leader_election.cc:302] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:43.599640 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 70886.8us
I20240507 08:06:43.530179 10260 raft_consensus.cc:2379] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 6: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:43.438162 13121 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.460027 13506 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:43.598709 10266 raft_consensus.cc:2454] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 7.
I20240507 08:06:43.477195 13377 raft_consensus.cc:729] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:43.443626 13205 raft_consensus.cc:504] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.480538 13564 leader_election.cc:288] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.610133 10155 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.450452 13461 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.585278 13412 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:43.553023 13562 leader_election.cc:288] T 7a181f4ae4d342129ad262e5545debc9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:43.555032 13475 consensus_queue.cc:237] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.565716 10089 catalog_manager.cc:5484] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 6 to 8, leader changed from bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 8 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:43.626662 10089 catalog_manager.cc:5484] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 6, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:43.627693 13377 consensus_queue.cc:260] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.620093 13157 raft_consensus.cc:3037] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:43.567441 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2275) took 1283 ms. Trace:
I20240507 08:06:43.629716 10188 rpcz_store.cc:276] 0507 08:06:42.284430 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:42.668708 (+384278us) service_pool.cc:225] Handling call
0507 08:06:42.669086 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.479259 (+810173us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.567241 (+ 87982us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540026dc54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:43.567392 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1719793920}
W20240507 08:06:43.578145 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 114291us
I20240507 08:06:43.616147 13487 consensus_queue.cc:1035] T 7dc98f346bff4d428582b1ad1dc0fc93 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20240507 08:06:43.630852 13461 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:43.589906 10304 leader_election.cc:302] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:43.615552 13414 consensus_queue.cc:1046] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20240507 08:06:43.531476 13569 raft_consensus.cc:2781] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20240507 08:06:43.525611 10306 leader_election.cc:302] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:43.636827 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 111388us
I20240507 08:06:43.602124 10266 raft_consensus.cc:1226] T 2a210876eb5b4a7ab6548839daccfe84 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:43.639434 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2031) took 1021 ms. Trace:
I20240507 08:06:43.639868 10266 rpcz_store.cc:276] 0507 08:06:42.618243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.601270 (+983027us) service_pool.cc:225] Handling call
0507 08:06:43.601909 (+ 639us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.639139 (+ 37230us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.639371 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:43.453264 10231 leader_election.cc:302] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:43.608517 13276 raft_consensus.cc:2726] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240507 08:06:43.537958 13488 consensus_queue.cc:1046] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20240507 08:06:43.545817 13361 consensus_queue.cc:1046] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20240507 08:06:43.586355 13413 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:43.623924 10185 raft_consensus.cc:1261] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20240507 08:06:43.659107 10185 pending_rounds.cc:76] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 3
I20240507 08:06:43.659430 10185 pending_rounds.cc:97] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20240507 08:06:43.659725 10185 raft_consensus.cc:2866] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:43.629354 13205 leader_election.cc:288] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.633487 10192 raft_consensus.cc:1226] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:43.633698 10304 leader_election.cc:302] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:43.550160 13147 consensus_queue.cc:260] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.615504 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.674372 10336 raft_consensus.cc:2379] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in current term 10: Already voted for candidate 88a26dac37fa4243b8c2b3d47f403488 in this term.
I20240507 08:06:43.595535 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
W20240507 08:06:43.641989 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 188884us
I20240507 08:06:43.556785 13548 consensus_queue.cc:237] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.652987 10188 raft_consensus.cc:1261] T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20240507 08:06:43.600816 13532 raft_consensus.cc:504] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.605662 13523 raft_consensus.cc:3037] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:43.668686 10260 raft_consensus.cc:1261] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240507 08:06:43.628635 10089 catalog_manager.cc:5484] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:43.646917 13564 raft_consensus.cc:2741] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20240507 08:06:43.653025 13374 raft_consensus.cc:2781] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20240507 08:06:43.647302 13442 raft_consensus.cc:3032] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Stepping down as leader of term 9
I20240507 08:06:43.687804 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:43.645496 13461 raft_consensus.cc:2781] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader election won for term 10
I20240507 08:06:43.679158 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:43.686154 10089 catalog_manager.cc:5484] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 5 to 6, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 6 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:43.692417 10231 leader_election.cc:302] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:43.691514 10089 catalog_manager.cc:5484] T 3df41f82bc854e7c9a22f3f345b03f92 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 7 to 8, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131). New cstate: current_term: 8 leader_uuid: "88a26dac37fa4243b8c2b3d47f403488" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
I20240507 08:06:43.698283 10089 catalog_manager.cc:5484] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 8 to 10, leader changed from 59382e6647c549d89d83c70233c0667e (127.8.213.129) to 5d6656a616b747d393ff3de5328d2540 (127.8.213.132). New cstate: current_term: 10 leader_uuid: "5d6656a616b747d393ff3de5328d2540" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } } }
I20240507 08:06:43.701829 13326 raft_consensus.cc:2781] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:43.702459 13326 raft_consensus.cc:686] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 LEADER]: Becoming Leader. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:43.704205 10264 raft_consensus.cc:2396] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240507 08:06:43.703498 13326 consensus_queue.cc:237] T 316d26400fa14c4b9e67f139ffeca8f1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.704864 13562 raft_consensus.cc:482] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:43.705410 13562 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.707284 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2026) took 1274 ms. Trace:
W20240507 08:06:43.707873 13198 consensus_meta.cc:287] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.745s user 0.006s sys 0.002s
I20240507 08:06:43.707789 10267 rpcz_store.cc:276] 0507 08:06:42.432987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.282497 (+849510us) service_pool.cc:225] Handling call
0507 08:06:43.282910 (+ 413us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.288457 (+ 5547us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.288506 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.289021 (+ 515us) log.cc:842] Serialized 26 byte log entry
0507 08:06:43.525032 (+236011us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:43.525090 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.525158 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:43.707030 (+181872us) raft_consensus.cc:1671] finished
0507 08:06:43.707058 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.707220 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9085,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":235612,"threads_started":1,"wal-append.queue_time_us":369591}
I20240507 08:06:43.709242 13121 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.677668 13319 raft_consensus.cc:2781] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader election won for term 7
I20240507 08:06:43.712399 10089 catalog_manager.cc:5484] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 reported cstate change: term changed from 9 to 10, leader changed from 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131) to bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130). New cstate: current_term: 10 leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } health_report { overall_health: UNKNOWN } } }
I20240507 08:06:43.662482 13346 raft_consensus.cc:2726] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20240507 08:06:43.715308 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2024) took 1346 ms. Trace:
I20240507 08:06:43.715766 10259 rpcz_store.cc:276] 0507 08:06:42.368692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.194766 (+826074us) service_pool.cc:225] Handling call
0507 08:06:43.195074 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.195343 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.195382 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.221512 (+ 26130us) log.cc:842] Serialized 26 byte log entry
0507 08:06:43.604414 (+382902us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:43.604521 (+ 107us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.604621 (+ 100us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:43.715035 (+110414us) raft_consensus.cc:1671] finished
0507 08:06:43.715062 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.715217 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":302269,"threads_started":1,"wal-append.queue_time_us":471537}
I20240507 08:06:43.658345 13566 consensus_queue.cc:237] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.725216 13513 consensus_queue.cc:1046] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20240507 08:06:43.673822 13375 consensus_queue.cc:1046] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20240507 08:06:43.668182 13364 consensus_queue.cc:1046] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20240507 08:06:43.726325 13513 raft_consensus.cc:3032] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Stepping down as leader of term 7
I20240507 08:06:43.726819 13513 raft_consensus.cc:729] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.727104 13364 raft_consensus.cc:3032] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Stepping down as leader of term 10
I20240507 08:06:43.727586 13364 raft_consensus.cc:729] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.727712 13513 consensus_queue.cc:260] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.729202 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8fba2019b474ce4b80d6b6d7c81a2ff" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.725723 13537 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:43.734151 10259 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20240507 08:06:43.748674 10259 raft_consensus.cc:1261] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240507 08:06:43.780133 10260 raft_consensus.cc:1226] T c6ac20a6d50a4700a577dfc18d71ceb2 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 LEADER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:43.757817 10333 raft_consensus.cc:2454] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
I20240507 08:06:43.784211 10259 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20240507 08:06:43.786439 10259 raft_consensus.cc:1261] T 8933e79c6c78466498a6d17d93ce7cff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:43.694480 13374 raft_consensus.cc:482] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:06:43.793931 10338 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:43.790884 10336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:43.792742 10259 raft_consensus.cc:3032] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Stepping down as leader of term 3
I20240507 08:06:43.791479 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1744) took 1172 ms. Trace:
I20240507 08:06:43.796334 10259 raft_consensus.cc:729] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:43.796416 10334 rpcz_store.cc:276] 0507 08:06:42.618836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.183497 (+564661us) service_pool.cc:225] Handling call
0507 08:06:43.791296 (+607799us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5400339ad4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.791424 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":873290112}
I20240507 08:06:43.797808 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:43.694298 13442 raft_consensus.cc:729] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:43.691673 13332 leader_election.cc:288] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:43.791926 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 120649us
I20240507 08:06:43.802479 10304 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.746635 13533 leader_election.cc:288] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.805521 10260 raft_consensus.cc:1261] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240507 08:06:43.809674 13545 consensus_queue.cc:1046] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:43.810062 10183 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55816 (request call id 1860) took 2423 ms. Trace:
I20240507 08:06:43.810945 10190 raft_consensus.cc:1261] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20240507 08:06:43.806407 10304 leader_election.cc:302] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.815570 10190 pending_rounds.cc:76] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Aborting all ops after (but not including) 3
I20240507 08:06:43.822237 10190 pending_rounds.cc:97] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20240507 08:06:43.822520 10190 raft_consensus.cc:2866] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:43.798122 10259 consensus_queue.cc:260] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.815448 10183 rpcz_store.cc:276] 0507 08:06:41.386483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.438344 (+ 51861us) service_pool.cc:225] Handling call
0507 08:06:43.809881 (+2371537us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54003b3654. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.810010 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4212101504}
I20240507 08:06:43.826887 12998 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240507 08:06:43.828270 12998 consensus_queue.cc:1035] T e0f47233c3be4fe99e097ade2aaec1e8 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240507 08:06:43.781404 13537 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.820924 13276 raft_consensus.cc:3032] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:43.830230 13276 raft_consensus.cc:729] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:43.830852 10337 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1735) took 1230 ms. Trace:
I20240507 08:06:43.831581 13276 consensus_queue.cc:260] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.831226 10337 rpcz_store.cc:276] 0507 08:06:42.600203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.213539 (+613336us) service_pool.cc:225] Handling call
0507 08:06:43.213869 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.215917 (+ 2048us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.215944 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.228306 (+ 12362us) log.cc:842] Serialized 26 byte log entry
0507 08:06:43.480099 (+251793us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:43.480136 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.480206 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:43.830609 (+350403us) raft_consensus.cc:1671] finished
0507 08:06:43.830633 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.830781 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1642,"raft.queue_time_us":5732,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":10502,"thread_start_us":220788,"threads_started":1,"wal-append.queue_time_us":561881}
I20240507 08:06:43.832891 13276 raft_consensus.cc:3037] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:43.825452 10259 raft_consensus.cc:3037] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Advancing to term 4
I20240507 08:06:43.838236 13527 raft_consensus.cc:504] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.839318 13157 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20240507 08:06:43.839407 10334 raft_consensus.cc:1226] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 6. Current term is 8. Ops: [6.4-6.4]
I20240507 08:06:43.800185 13374 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:43.831616 13537 leader_election.cc:288] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.817363 10304 leader_election.cc:302] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.843075 13332 raft_consensus.cc:2726] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:43.843461 13545 consensus_queue.cc:1225] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer bcf08c8276ba4a9ba646ad0f7090fcd8 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20240507 08:06:43.843984 13527 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:43.805995 13320 consensus_queue.cc:1046] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.039s
I20240507 08:06:43.843456 10304 leader_election.cc:302] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 5d6656a616b747d393ff3de5328d2540
I20240507 08:06:43.761384 13521 consensus_queue.cc:1035] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20240507 08:06:43.793921 13319 raft_consensus.cc:686] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [term 7 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.899578 13524 raft_consensus.cc:3032] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:43.900008 13524 raft_consensus.cc:729] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: bcf08c8276ba4a9ba646ad0f7090fcd8, State: Running, Role: LEADER
I20240507 08:06:43.899900 13319 consensus_queue.cc:237] T ad8dd18470a542d3bec18f13b451878d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.901815 13532 leader_election.cc:288] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:43.859882 10260 raft_consensus.cc:1261] T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:43.867494 13418 raft_consensus.cc:3032] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 LEADER]: Stepping down as leader of term 10
I20240507 08:06:43.868532 13115 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:43.872045 13426 consensus_queue.cc:260] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.877751 10259 raft_consensus.cc:1261] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20240507 08:06:43.880088 13399 raft_consensus.cc:3032] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Stepping down as leader of term 9
W20240507 08:06:43.882561 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 415270us
I20240507 08:06:43.917352 10259 pending_rounds.cc:76] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 2
I20240507 08:06:43.917775 10259 pending_rounds.cc:97] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20240507 08:06:43.918201 10259 raft_consensus.cc:2866] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:43.918856 10232 leader_election.cc:398] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9.
I20240507 08:06:43.919272 10232 leader_election.cc:401] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20240507 08:06:43.728495 13364 consensus_queue.cc:260] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.888370 13442 consensus_queue.cc:260] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.809695 13337 raft_consensus.cc:2726] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20240507 08:06:43.889089 13567 log.cc:924] Time spent T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.608s user 0.003s sys 0.000s
I20240507 08:06:43.822867 13506 consensus_queue.cc:1035] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20240507 08:06:43.799679 13519 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:43.890086 13320 raft_consensus.cc:3032] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Stepping down as leader of term 10
I20240507 08:06:43.893913 13198 raft_consensus.cc:2781] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20240507 08:06:43.894569 13469 consensus_meta.cc:287] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.727s user 0.010s sys 0.000s
I20240507 08:06:43.895352 10267 pending_rounds.cc:76] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting all ops after (but not including) 6
I20240507 08:06:43.928764 10267 pending_rounds.cc:97] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20240507 08:06:43.929121 10267 raft_consensus.cc:2866] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240507 08:06:43.897418 10264 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 2036) took 1018 ms. Trace:
I20240507 08:06:43.897745 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1741) took 1295 ms. Trace:
I20240507 08:06:43.900874 13524 consensus_queue.cc:260] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:43.916251 13115 raft_consensus.cc:3037] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:43.916301 13418 raft_consensus.cc:729] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:43.920380 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 60746.4us
I20240507 08:06:43.930166 10264 rpcz_store.cc:276] 0507 08:06:42.879461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.051208 (+171747us) service_pool.cc:225] Handling call
0507 08:06:43.897213 (+846005us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b54001d0e54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.897360 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1236448640}
I20240507 08:06:43.929924 10333 rpcz_store.cc:276] 0507 08:06:42.601931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:42.973766 (+371835us) service_pool.cc:225] Handling call
0507 08:06:43.897599 (+923833us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b5400378a54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:43.897693 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1626555648}
I20240507 08:06:43.922008 13399 raft_consensus.cc:729] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:43.933197 10340 raft_consensus.cc:1261] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20240507 08:06:43.928421 13198 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:43.903940 10265 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 525) took 1472 ms. Trace:
I20240507 08:06:43.953955 10265 rpcz_store.cc:276] 0507 08:06:42.431655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.279876 (+848221us) service_pool.cc:225] Handling call
0507 08:06:43.280108 (+ 232us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.280261 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 8
0507 08:06:43.570888 (+290627us) log.cc:842] Serialized 14 byte log entry
0507 08:06:43.903522 (+332634us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:43.903587 (+ 65us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:43.903647 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.903733 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:43.903869 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6586,"prepare.queue_time_us":545222,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":631,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":15785,"thread_start_us":620868,"threads_started":2}
I20240507 08:06:43.947856 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:43.948194 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 336690us
I20240507 08:06:43.957944 10155 leader_election.cc:302] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.959754 10333 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:43.926862 13320 raft_consensus.cc:729] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:43.972451 10231 leader_election.cc:302] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:43.929438 13426 raft_consensus.cc:3037] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Advancing to term 7
I20240507 08:06:43.978454 10155 leader_election.cc:302] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:43.947611 13364 raft_consensus.cc:3037] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:43.979570 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:43.982992 10138 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica bcf08c8276ba4a9ba646ad0f7090fcd8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } }
W20240507 08:06:43.928403 13455 consensus_queue.cc:1159] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" responder_term: 10 status { last_received { term: 10 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 10 index: 6 } }
I20240507 08:06:43.931735 13523 raft_consensus.cc:2726] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795) with higher term. Message: Invalid argument: T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 8. Current term is 9.
I20240507 08:06:43.926609 13527 consensus_queue.cc:1046] T 2a210876eb5b4a7ab6548839daccfe84 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20240507 08:06:43.983788 10265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19878aa675724839ac9bb59333dee2d9" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:44.030550 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95da5fd11446429481eb60f5a0c8158e" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:44.032158 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:44.035396 13320 consensus_queue.cc:260] T a8fba2019b474ce4b80d6b6d7c81a2ff P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.948832 13570 consensus_queue.cc:1035] T 67fb9faf0618484ab4dd4cfb90e69c28 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20240507 08:06:44.031894 13157 consensus_queue.cc:1035] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20240507 08:06:44.057745 13418 consensus_queue.cc:260] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:43.968400 13198 raft_consensus.cc:3037] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:44.042474 13441 consensus_queue.cc:1035] T 8933e79c6c78466498a6d17d93ce7cff P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240507 08:06:44.046294 13523 raft_consensus.cc:2726] T a8fba2019b474ce4b80d6b6d7c81a2ff P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20240507 08:06:44.046532 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 74214.4us
I20240507 08:06:43.979521 13310 raft_consensus.cc:2726] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20240507 08:06:43.978837 13361 raft_consensus.cc:482] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:43.981058 13492 raft_consensus.cc:3032] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Stepping down as leader of term 9
I20240507 08:06:43.953166 13469 consensus_queue.cc:1035] T aa60df83ab754c7e9229c7775883d22d P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.117s
I20240507 08:06:44.031283 13524 raft_consensus.cc:3037] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:44.033689 13121 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.031883 13533 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20240507 08:06:44.069211 13532 raft_consensus.cc:482] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:44.086524 13576 log.cc:924] Time spent T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20240507 08:06:44.086097 13506 consensus_queue.cc:1035] T e0a774a8075d4c689b047bb943c78c50 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20240507 08:06:44.086434 13332 consensus_queue.cc:1035] T 765054c26d4a4181b4fb31891d033bb1 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20240507 08:06:44.042004 13346 raft_consensus.cc:2726] T 765054c26d4a4181b4fb31891d033bb1 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240507 08:06:44.117065 13374 raft_consensus.cc:504] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.157145 10265 raft_consensus.cc:3037] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:44.195255 13492 raft_consensus.cc:729] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
W20240507 08:06:44.097209 10158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 482254us
I20240507 08:06:44.182716 10231 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488
I20240507 08:06:44.183702 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:44.153110 9046 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240507 08:06:44.162837 10192 raft_consensus.cc:1226] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Rejecting Update request from peer 88a26dac37fa4243b8c2b3d47f403488 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:44.167035 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d26fdad3816d470a8a7537d14838ac68" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:44.248078 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82c5eb4243284619ad55e8a724ff32de" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:44.167765 10304 leader_election.cc:302] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 59382e6647c549d89d83c70233c0667e, bcf08c8276ba4a9ba646ad0f7090fcd8
W20240507 08:06:44.250546 10304 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 82913.3us
I20240507 08:06:44.252159 10155 leader_election.cc:302] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:44.253566 10155 leader_election.cc:302] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:44.254853 13589 raft_consensus.cc:2726] T d26fdad3816d470a8a7537d14838ac68 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20240507 08:06:44.255504 13589 raft_consensus.cc:2726] T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:44.257081 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:44.240360 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 57788.4us
I20240507 08:06:44.249362 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ae3c4ecd5e46ec9b5cf741db56c101" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:44.261807 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:44.262415 10264 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:44.269264 10155 leader_election.cc:302] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:44.278434 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1746) took 1596 ms. Trace:
I20240507 08:06:44.180790 13461 raft_consensus.cc:686] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
W20240507 08:06:44.189797 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 362499us
I20240507 08:06:44.068332 13399 consensus_queue.cc:260] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.192041 13086 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:44.200984 13412 consensus_meta.cc:287] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.563s user 0.008s sys 0.000s
I20240507 08:06:44.204227 13147 raft_consensus.cc:3037] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:44.204882 13361 raft_consensus.cc:504] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.190853 13537 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.207908 10190 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2260) took 2263 ms. Trace:
I20240507 08:06:44.319914 10190 rpcz_store.cc:276] 0507 08:06:41.944231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:42.291801 (+347570us) service_pool.cc:225] Handling call
0507 08:06:43.810635 (+1518834us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.823007 (+ 12372us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.207728 (+384721us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54003b3650. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:44.207852 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3494038400}
I20240507 08:06:44.208566 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 444) took 1381 ms. Trace:
I20240507 08:06:44.178869 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64632a3e07184039950f3ef7fe699cb3" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:44.212011 13564 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.276319 10265 raft_consensus.cc:2454] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
I20240507 08:06:44.322376 10183 raft_consensus.cc:3037] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:44.322130 10335 rpcz_store.cc:276] 0507 08:06:42.827602 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.329551 (+501949us) service_pool.cc:225] Handling call
0507 08:06:43.329834 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.526513 (+196679us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.526548 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:43.526583 (+ 35us) raft_consensus.cc:1631] Marking committed up to 9
0507 08:06:43.526612 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.208354 (+681742us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54003c4454. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.208399 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.208514 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":452109184}
I20240507 08:06:44.323179 10265 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240507 08:06:44.278805 10341 rpcz_store.cc:276] 0507 08:06:42.681999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.258330 (+576331us) service_pool.cc:225] Handling call
0507 08:06:43.280012 (+ 21682us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.280183 (+ 171us) raft_consensus.cc:1520] Early marking committed up to index 4
0507 08:06:43.524789 (+244606us) log.cc:842] Serialized 14 byte log entry
0507 08:06:44.278100 (+753311us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:44.278162 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:44.278205 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.278280 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.278383 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":400079,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":107684,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":2541,"thread_start_us":993524,"threads_started":3,"wal-append.queue_time_us":652948}
I20240507 08:06:44.313503 10333 raft_consensus.cc:1226] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:44.302856 13421 raft_consensus.cc:2726] T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:44.307907 13276 raft_consensus.cc:2726] T 63ae3c4ecd5e46ec9b5cf741db56c101 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240507 08:06:44.218364 13121 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.335201 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 446) took 1036 ms. Trace:
I20240507 08:06:44.335693 10335 rpcz_store.cc:276] 0507 08:06:43.298861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.334288 (+1035427us) service_pool.cc:225] Handling call
0507 08:06:44.334610 (+ 322us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.334793 (+ 183us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.334831 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:44.334883 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:44.334926 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.335015 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.335132 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:44.338860 10265 raft_consensus.cc:1261] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20240507 08:06:44.345707 10264 raft_consensus.cc:2396] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 88a26dac37fa4243b8c2b3d47f403488 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20240507 08:06:44.346786 10264 raft_consensus.cc:1261] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:44.223410 13532 raft_consensus.cc:504] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.351907 10335 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
W20240507 08:06:44.324630 13488 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.016s sys 0.063s
I20240507 08:06:44.368279 13533 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.415153 10258 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 523) took 2015 ms. Trace:
I20240507 08:06:44.370265 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2029) took 1767 ms. Trace:
I20240507 08:06:44.323881 13523 raft_consensus.cc:2726] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:44.416148 10263 rpcz_store.cc:276] 0507 08:06:42.602525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.451463 (+848938us) service_pool.cc:225] Handling call
0507 08:06:43.478462 (+ 26999us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.478677 (+ 215us) raft_consensus.cc:1520] Early marking committed up to index 4
0507 08:06:43.938993 (+460316us) log.cc:842] Serialized 14 byte log entry
0507 08:06:44.369484 (+430491us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:44.369552 (+ 68us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:44.369950 (+ 398us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.370053 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.370199 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36048,"prepare.queue_time_us":349142,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"thread_start_us":796540,"threads_started":2}
I20240507 08:06:44.415647 10258 rpcz_store.cc:276] 0507 08:06:42.399321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.235116 (+835795us) service_pool.cc:225] Handling call
0507 08:06:43.235397 (+ 281us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.235715 (+ 318us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.235741 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.236112 (+ 371us) log.cc:842] Serialized 26 byte log entry
0507 08:06:43.514566 (+278454us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:43.959279 (+444713us) log.cc:842] Serialized 14 byte log entry
0507 08:06:44.414866 (+455587us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.414935 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:44.414975 (+ 40us) raft_consensus.cc:1671] finished
0507 08:06:44.414987 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.415093 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43482,"prepare.queue_time_us":313383,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":86517,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":15592,"thread_start_us":1055625,"threads_started":3,"wal-append.queue_time_us":432875}
I20240507 08:06:44.430045 10183 raft_consensus.cc:2454] T 64632a3e07184039950f3ef7fe699cb3 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
W20240507 08:06:44.412724 13488 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s user 0.016s sys 0.063s
W20240507 08:06:44.420506 13548 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.006s sys 0.022s
I20240507 08:06:44.444959 10264 raft_consensus.cc:1226] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20240507 08:06:44.312880 13086 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
W20240507 08:06:44.512535 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 252656us
I20240507 08:06:44.318377 13412 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:44.328773 13537 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.329135 13564 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:44.511468 13548 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.007s sys 0.022s
W20240507 08:06:44.544986 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1cd78797b4574a049447c358499ad846, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.329344 13399 raft_consensus.cc:3037] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:44.415997 13276 consensus_queue.cc:1035] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20240507 08:06:44.548758 13596 raft_consensus.cc:2726] T 95da5fd11446429481eb60f5a0c8158e P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:44.406631 13310 consensus_queue.cc:1035] T 64f8f2cf89ed4b99822bc4389d7a842f P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
W20240507 08:06:44.558892 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 276021us
I20240507 08:06:44.420138 13533 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.367532 13492 consensus_queue.cc:260] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.557637 10261 raft_consensus.cc:1226] T bfb70b9953fd486ca5f88b5999dd26a0 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:44.512315 13332 raft_consensus.cc:482] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:44.606768 13426 consensus_meta.cc:287] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.624s user 0.008s sys 0.001s
W20240507 08:06:44.674124 13602 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.306s user 0.000s sys 0.000s
I20240507 08:06:44.549587 13319 raft_consensus.cc:482] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:44.555276 10907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 173262us
I20240507 08:06:44.616113 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1748) took 1673 ms. Trace:
I20240507 08:06:44.556473 10266 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2035) took 1677 ms. Trace:
I20240507 08:06:44.626176 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1757) took 1461 ms. Trace:
I20240507 08:06:44.618992 13414 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.548282 10335 raft_consensus.cc:3037] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:44.612079 10306 leader_election.cc:302] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters:
W20240507 08:06:44.674958 13597 log.cc:924] Time spent T 03d2e621edea41ae97e79e36e4dc0365 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20240507 08:06:44.676362 13086 leader_election.cc:288] T 2e5fd8588b9240c38e1f3c75cf342156 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:44.676642 13533 leader_election.cc:288] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:44.677826 13617 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.678153 13611 raft_consensus.cc:482] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.677729 13332 raft_consensus.cc:504] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.645373 13375 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20240507 08:06:44.605917 13610 raft_consensus.cc:482] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:44.683137 13412 leader_election.cc:288] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:44.684057 13564 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:44.686200 13595 consensus_queue.cc:1035] T 82c5eb4243284619ad55e8a724ff32de P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240507 08:06:44.550081 13461 consensus_queue.cc:237] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.687695 13374 leader_election.cc:288] T 534af8458dc14ec49ca7308cf02b685c P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:44.688313 13537 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:44.688969 13121 leader_election.cc:288] T aa60df83ab754c7e9229c7775883d22d P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:44.691535 13532 leader_election.cc:288] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:44.692808 13361 leader_election.cc:288] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:44.695318 13569 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:44.698271 13592 log.cc:924] Time spent T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.373s user 0.001s sys 0.000s
I20240507 08:06:44.631134 13590 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:44.727382 13198 consensus_meta.cc:287] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.537s user 0.010s sys 0.000s
I20240507 08:06:44.734544 13596 raft_consensus.cc:2781] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader election won for term 8
I20240507 08:06:44.768280 13198 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:44.729400 13414 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:44.770362 13198 leader_election.cc:288] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 election: Requested vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:44.771019 13414 leader_election.cc:288] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:44.768522 13596 raft_consensus.cc:686] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:44.772485 13596 consensus_queue.cc:237] T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:44.774964 13582 log.cc:924] Time spent T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.706s user 0.002s sys 0.000s
I20240507 08:06:44.775305 13590 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:44.777259 13587 log.cc:924] Time spent T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.578s user 0.000s sys 0.001s
I20240507 08:06:44.739617 13617 raft_consensus.cc:504] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:44.780046 13585 log.cc:924] Time spent T f0778bf2d68148c495412379f7818b28 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.563s user 0.001s sys 0.000s
I20240507 08:06:44.780503 13617 leader_election.cc:288] T 8f7d05e73be847e99048092c9570faed P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:44.781801 13443 consensus_queue.cc:260] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:44.724196 13319 raft_consensus.cc:504] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.739001 13611 raft_consensus.cc:504] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.764401 13569 raft_consensus.cc:3037] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:44.777333 13590 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:44.744124 13610 raft_consensus.cc:504] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.797369 13611 leader_election.cc:288] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:44.828357 13604 log.cc:924] Time spent T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.071s user 0.000s sys 0.001s
I20240507 08:06:44.844604 13610 leader_election.cc:288] T 971f59883ca347648664bc7f2fcaf773 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
W20240507 08:06:44.845118 13593 log.cc:924] Time spent T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.515s user 0.000s sys 0.006s
W20240507 08:06:44.551318 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1cd78797b4574a049447c358499ad846, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:44.874758 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2310) took 1199 ms. Trace:
W20240507 08:06:44.552033 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 204128us
I20240507 08:06:44.875608 10185 rpcz_store.cc:276] 0507 08:06:43.674873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.710172 (+ 35299us) service_pool.cc:225] Handling call
0507 08:06:43.710534 (+ 362us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.759024 (+ 48490us) raft_consensus.cc:1520] Early marking committed up to index 4
0507 08:06:43.759067 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.759586 (+ 519us) log.cc:842] Serialized 26 byte log entry
0507 08:06:44.157344 (+397758us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:44.157386 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.157428 (+ 42us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:44.873758 (+716330us) raft_consensus.cc:1671] finished
0507 08:06:44.873847 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.874388 (+ 541us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18521,"raft.queue_time_us":48210,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":8603,"thread_start_us":397490,"threads_started":1,"wal-append.queue_time_us":1032668}
I20240507 08:06:44.878973 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3df41f82bc854e7c9a22f3f345b03f92" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:44.879797 10183 raft_consensus.cc:1261] T e57f967aaaf34bd5acd85d28edeb4abe P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:44.880364 10185 raft_consensus.cc:2454] T 3df41f82bc854e7c9a22f3f345b03f92 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
I20240507 08:06:44.881907 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:44.882089 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 339062us
I20240507 08:06:44.883472 10185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a181f4ae4d342129ad262e5545debc9" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:44.885118 10185 raft_consensus.cc:1226] T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e [term 6 LEADER]: Rejecting Update request from peer bcf08c8276ba4a9ba646ad0f7090fcd8 for earlier term 5. Current term is 6. Ops: []
I20240507 08:06:44.886813 10185 raft_consensus.cc:1261] T c6ac20a6d50a4700a577dfc18d71ceb2 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20240507 08:06:44.557711 10188 raft_consensus.cc:1226] T 5d0e1a7b6db34cea8e152b755b52aa96 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:44.890709 10187 raft_consensus.cc:1261] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Refusing update from remote peer 5d6656a616b747d393ff3de5328d2540: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
W20240507 08:06:44.892073 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 374589us
I20240507 08:06:44.892426 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:44.893558 10190 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 10.
I20240507 08:06:44.894331 10189 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2298) took 1816 ms. Trace:
I20240507 08:06:44.895618 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2038) took 1567 ms. Trace:
I20240507 08:06:44.896095 10267 rpcz_store.cc:276] 0507 08:06:43.327720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.732770 (+405050us) service_pool.cc:225] Handling call
0507 08:06:43.894997 (+162227us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b58002f0198. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:06:43.895183 (+ 186us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.929639 (+ 34456us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.929669 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.930125 (+ 456us) log.cc:842] Serialized 26 byte log entry
0507 08:06:44.157588 (+227463us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:44.157646 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.157708 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:44.894936 (+737228us) raft_consensus.cc:1671] finished
0507 08:06:44.894973 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.895333 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3573120,"thread_start_us":227016,"threads_started":1,"wal-append.queue_time_us":377878}
I20240507 08:06:44.716905 10266 rpcz_store.cc:276] 0507 08:06:42.879091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.640939 (+761848us) service_pool.cc:225] Handling call
0507 08:06:43.641267 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.641498 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 5
0507 08:06:44.068223 (+426725us) log.cc:842] Serialized 14 byte log entry
0507 08:06:44.251192 (+182969us) log.cc:842] Serialized 14 byte log entry
0507 08:06:44.555672 (+304480us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:44.555878 (+ 206us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:44.555964 (+ 86us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.556132 (+ 168us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.556352 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3135,"prepare.queue_time_us":180758,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":115019,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":13785,"spinlock_wait_cycles":7823744,"thread_start_us":883399,"threads_started":3,"wal-append.queue_time_us":575795}
I20240507 08:06:44.894842 10189 rpcz_store.cc:276] 0507 08:06:43.103325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.223755 (+120430us) service_pool.cc:225] Handling call
0507 08:06:43.242560 (+ 18805us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.242898 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.242924 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.253139 (+ 10215us) log.cc:842] Serialized 26 byte log entry
0507 08:06:43.686048 (+432909us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:43.686082 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:43.686139 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:44.774449 (+1088310us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:44.893752 (+119303us) raft_consensus.cc:1671] finished
0507 08:06:44.893806 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.894135 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32850,"raft.queue_time_us":6561,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":5587,"spinlock_wait_cycles":358714624,"thread_start_us":371929,"threads_started":1,"wal-append.queue_time_us":685575}
I20240507 08:06:44.902463 10260 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39118 (request call id 532) took 1066 ms. Trace:
I20240507 08:06:44.716032 10333 rpcz_store.cc:276] 0507 08:06:43.164535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.258121 (+1093586us) service_pool.cc:225] Handling call
0507 08:06:44.258444 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.331804 (+ 73360us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.625856 (+294052us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b5400049f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.626102 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":123724160}
I20240507 08:06:44.903043 10260 rpcz_store.cc:276] 0507 08:06:43.835389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.903315 (+ 67926us) service_pool.cc:225] Handling call
0507 08:06:43.925068 (+ 21753us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.949377 (+ 24309us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.949419 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:43.949909 (+ 490us) log.cc:842] Serialized 26 byte log entry
0507 08:06:44.332972 (+383063us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:44.333077 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.333137 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:44.901854 (+568717us) raft_consensus.cc:1671] finished
0507 08:06:44.901901 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.902213 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88428,"raft.queue_time_us":25058,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":24461,"thread_start_us":382696,"threads_started":1,"wal-append.queue_time_us":788270}
I20240507 08:06:44.905851 10304 leader_election.cc:302] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters: 59382e6647c549d89d83c70233c0667e
I20240507 08:06:44.906628 10261 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 2044) took 2668 ms. Trace:
I20240507 08:06:44.906996 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 450) took 1271 ms. Trace:
I20240507 08:06:44.907063 10261 rpcz_store.cc:276] 0507 08:06:42.238662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.190241 (+951579us) service_pool.cc:225] Handling call
0507 08:06:43.231768 (+ 41527us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.644956 (+1413188us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.906109 (+261153us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400117954. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.906457 (+ 348us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3050220288}
I20240507 08:06:44.907717 10333 rpcz_store.cc:276] 0507 08:06:43.635662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.905211 (+1269549us) service_pool.cc:225] Handling call
0507 08:06:44.905987 (+ 776us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.906457 (+ 470us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.906486 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:44.906543 (+ 57us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:44.906580 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.906658 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.906759 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:44.728654 10306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 116739us
I20240507 08:06:44.910879 10333 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 451) took 1240 ms. Trace:
I20240507 08:06:44.914641 10190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e"
I20240507 08:06:44.917073 10190 raft_consensus.cc:3037] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:44.917830 10188 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 428) took 1236 ms. Trace:
I20240507 08:06:44.918725 10187 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 426) took 1597 ms. Trace:
I20240507 08:06:44.919196 10187 rpcz_store.cc:276] 0507 08:06:43.320976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.520332 (+199356us) service_pool.cc:225] Handling call
0507 08:06:43.550872 (+ 30540us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.891398 (+1340526us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.918415 (+ 27017us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54003b4f54. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.918636 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3080328576}
I20240507 08:06:44.918272 10188 rpcz_store.cc:276] 0507 08:06:43.681394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.695561 (+ 14167us) service_pool.cc:225] Handling call
0507 08:06:43.695901 (+ 340us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.890105 (+1194204us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.917360 (+ 27255us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54002dc3d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.917626 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1982541056}
I20240507 08:06:44.920676 10187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:44.915932 10332 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1732) took 3295 ms (client timeout 3040 ms). Trace:
I20240507 08:06:44.921641 10333 rpcz_store.cc:276] 0507 08:06:43.670286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.909305 (+1239019us) service_pool.cc:225] Handling call
0507 08:06:44.909920 (+ 615us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:44.910419 (+ 499us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.910460 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:44.910550 (+ 90us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:44.910603 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.910682 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.910819 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:44.893502 10186 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1896) took 1584 ms. Trace:
I20240507 08:06:44.916114 10189 raft_consensus.cc:1261] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240507 08:06:44.923233 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:44.714991 10334 rpcz_store.cc:276] 0507 08:06:42.942444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.821687 (+879243us) service_pool.cc:225] Handling call
0507 08:06:43.822050 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:43.839940 (+ 17890us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.615762 (+775822us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b540014e1d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.616035 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39443712}
I20240507 08:06:44.629324 10338 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2172) took 2559 ms. Trace:
W20240507 08:06:44.921716 10332 rpcz_store.cc:269] 0507 08:06:41.620167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:41.668823 (+ 48656us) service_pool.cc:225] Handling call
0507 08:06:44.915416 (+3246593us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b54002c74d4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e2b4fe 00007fcf95d699c7 00007fcf8f44dc14 00007fcf8f44dba2 00007fcf8f44dad1 00007fcf8f44da4c 00007fcf8f44cd02 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:44.915665 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5600433024}
I20240507 08:06:44.926561 10337 raft_consensus.cc:3032] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Stepping down as leader of term 6
I20240507 08:06:44.926802 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2188) took 2006 ms. Trace:
I20240507 08:06:44.923039 10186 rpcz_store.cc:276] 0507 08:06:43.309224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.450906 (+141682us) service_pool.cc:225] Handling call
0507 08:06:43.520455 (+ 69549us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.592939 (+ 72484us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.892977 (+1300038us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b5400046850. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:44.893283 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":94779776}
I20240507 08:06:44.927841 10341 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2192) took 1536 ms. Trace:
I20240507 08:06:44.931528 10339 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1747) took 2026 ms. Trace:
I20240507 08:06:44.936436 10187 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
W20240507 08:06:44.706048 10908 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 380412us
I20240507 08:06:44.881464 10183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e83f7f3abec42d2bc76c5648cfd0aa3" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
W20240507 08:06:45.060608 10231 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 161486us
I20240507 08:06:45.088666 10334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:45.093719 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dc98f346bff4d428582b1ad1dc0fc93" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:45.095132 13332 leader_election.cc:288] T 64f8f2cf89ed4b99822bc4389d7a842f P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:45.097537 13617 raft_consensus.cc:482] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.097776 13631 raft_consensus.cc:482] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.098599 13455 raft_consensus.cc:2781] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Leader election won for term 9
I20240507 08:06:45.080026 10340 rpcz_store.cc:276] 0507 08:06:42.920088 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.568447 (+648359us) service_pool.cc:225] Handling call
0507 08:06:43.568651 (+ 204us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.950209 (+381558us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.926292 (+976083us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b540034eed4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:44.926610 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":837780224}
I20240507 08:06:45.100221 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67fb9faf0618484ab4dd4cfb90e69c28" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:45.079560 10337 raft_consensus.cc:729] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:45.080610 10339 rpcz_store.cc:276] 0507 08:06:42.904733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.391580 (+486847us) service_pool.cc:225] Handling call
0507 08:06:43.395734 (+ 4154us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.471900 (+ 76166us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.204784 (+732884us) spinlock_profiling.cc:243] Waited 42.9 ms on lock 0x7b54000ff8a0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dacd30 00007fcf90db7973 00007fcf90e2ad92 00007fcf90e27176 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782
0507 08:06:44.930893 (+726109us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b54000ff8a0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90dbcf0d 00007fcf90e2ade7 00007fcf90e27176 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782
0507 08:06:44.931291 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":168831616}
I20240507 08:06:45.101212 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:45.080377 10341 rpcz_store.cc:276] 0507 08:06:43.391191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.326073 (+934882us) service_pool.cc:225] Handling call
0507 08:06:44.326578 (+ 505us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:44.326845 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.326881 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:44.374128 (+ 47247us) log.cc:842] Serialized 26 byte log entry
0507 08:06:44.744032 (+369904us) raft_consensus.cc:1631] Marking committed up to 8
0507 08:06:44.744071 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.744135 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:44.927311 (+183176us) raft_consensus.cc:1671] finished
0507 08:06:44.927351 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:44.927616 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":306251,"threads_started":1,"wal-append.queue_time_us":392812}
W20240507 08:06:45.112411 13527 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.034s user 0.020s sys 0.024s
I20240507 08:06:45.112720 10333 raft_consensus.cc:2454] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 6.
W20240507 08:06:45.112998 13527 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.077s user 0.021s sys 0.024s
I20240507 08:06:45.113684 13590 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.113829 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:45.114614 13590 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:45.115495 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:45.116282 10333 raft_consensus.cc:2454] T 10eece2dd4a64e3aa99d91fadc43df89 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 8.
I20240507 08:06:45.117195 10333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
W20240507 08:06:45.120488 13147 consensus_meta.cc:287] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.799s user 0.010s sys 0.001s
I20240507 08:06:45.123612 13319 leader_election.cc:288] T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:45.125212 13595 raft_consensus.cc:482] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
W20240507 08:06:45.126030 10906 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 168530us
I20240507 08:06:45.126497 13519 consensus_queue.cc:1035] T 292ec14f14384ba1bc9ee81980e1eb2a P 88a26dac37fa4243b8c2b3d47f403488 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:45.130527 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d0e1a7b6db34cea8e152b755b52aa96" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:45.132375 13441 raft_consensus.cc:482] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:45.133291 13414 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.133960 10939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f101fe713f47b1b84bcaeb54f7aead" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:45.135051 10938 raft_consensus.cc:1226] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [term 7 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 6. Current term is 7. Ops: []
I20240507 08:06:45.135701 10189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:45.136212 13532 consensus_queue.cc:1046] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:45.135811 13632 consensus_queue.cc:1046] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240507 08:06:45.137327 13488 raft_consensus.cc:482] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.188400 13620 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.188570 13488 raft_consensus.cc:504] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:45.189786 13617 raft_consensus.cc:504] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:45.190527 13086 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.136893 13631 raft_consensus.cc:504] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.191341 13086 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.192734 13617 leader_election.cc:288] T e940ba6acb8d4782a4c921b50b882f91 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:45.193205 13642 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.193454 13628 raft_consensus.cc:482] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.193941 13642 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.194219 13628 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.088570 13121 consensus_queue.cc:1035] T e57f967aaaf34bd5acd85d28edeb4abe P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.022s
I20240507 08:06:45.135705 10338 rpcz_store.cc:276] 0507 08:06:42.069777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.194427 (+1124650us) service_pool.cc:225] Handling call
0507 08:06:43.791545 (+597118us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:43.795307 (+ 3762us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.629082 (+833775us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b5400339ad0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e26c59 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006 00007fcf8f24d9af
0507 08:06:44.629250 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1373275904}
I20240507 08:06:45.152040 10190 raft_consensus.cc:2454] T 534af8458dc14ec49ca7308cf02b685c P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
I20240507 08:06:45.157878 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7c60767d74458094c1d86a49e87980" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:45.157491 13595 raft_consensus.cc:504] T d26fdad3816d470a8a7537d14838ac68 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.159121 10334 raft_consensus.cc:2454] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:45.137993 13361 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.138031 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f4b60e5e6294639a468641f215329a5" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540"
I20240507 08:06:45.189339 13620 raft_consensus.cc:504] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:45.641491 10335 consensus_meta.cc:287] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 0.635s user 0.004s sys 0.008s
I20240507 08:06:45.218847 13638 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.647590 10335 raft_consensus.cc:1261] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20240507 08:06:45.220891 10339 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "316d26400fa14c4b9e67f139ffeca8f1"
dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
mode: GRACEFUL
new_leader_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
from {username='slave'} at 127.0.0.1:40074
I20240507 08:06:45.648851 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2195) took 2057 ms. Trace:
I20240507 08:06:45.649121 10186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "59382e6647c549d89d83c70233c0667e" is_pre_election: true
I20240507 08:06:45.650000 10186 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 10.
I20240507 08:06:45.647974 10192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2315) took 1486 ms. Trace:
I20240507 08:06:45.221244 10261 raft_consensus.cc:3037] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Advancing to term 9
I20240507 08:06:45.362838 13533 raft_consensus.cc:482] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader bcf08c8276ba4a9ba646ad0f7090fcd8)
I20240507 08:06:45.549698 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e57f967aaaf34bd5acd85d28edeb4abe" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:45.186825 10263 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.000s sys 0.014s
I20240507 08:06:45.218533 10940 raft_consensus.cc:2379] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 9: Already voted for candidate 5d6656a616b747d393ff3de5328d2540 in this term.
I20240507 08:06:45.221446 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10eece2dd4a64e3aa99d91fadc43df89" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:45.438498 10336 raft_consensus.cc:1226] T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 LEADER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: [9.7-9.7]
W20240507 08:06:45.372820 10230 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.004s sys 0.012s
I20240507 08:06:45.649616 10335 rpcz_store.cc:276] 0507 08:06:43.591930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.547718 (+955788us) service_pool.cc:225] Handling call
0507 08:06:44.548026 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:45.648477 (+1100451us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.648780 (+ 303us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:45.642760 13569 consensus_meta.cc:287] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 0.839s user 0.006s sys 0.009s
I20240507 08:06:45.489480 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:45.438529 13649 log.cc:924] Time spent T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.057s user 0.000s sys 0.002s
W20240507 08:06:45.527750 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 122656us
W20240507 08:06:45.680666 10263 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.165s user 0.009s sys 0.063s
W20240507 08:06:45.324303 10909 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.020s sys 0.011s
W20240507 08:06:45.701915 10230 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.251s user 0.007s sys 0.027s
I20240507 08:06:45.706701 13418 raft_consensus.cc:3037] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Advancing to term 11
I20240507 08:06:45.707116 13442 raft_consensus.cc:3037] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:45.222203 13414 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:45.711890 10258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e5fd8588b9240c38e1f3c75cf342156" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:45.713059 13414 leader_election.cc:288] T 9800706282e543be9f2ee3473da2b89f P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:45.339988 13590 leader_election.cc:288] T 5a168a24022a4678bd74fa028a765700 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:45.553107 13121 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.714283 13121 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.716327 13121 leader_election.cc:288] T 7a123878203f45319def3c4246b9fe30 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:45.325033 13524 consensus_meta.cc:287] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8: Time spent flushing consensus metadata: real 1.108s user 0.015s sys 0.000s
W20240507 08:06:45.716923 13600 log.cc:924] Time spent T 4cdcc48b264f4a9eb3eabce20e5bbd6c P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.950s user 0.000s sys 0.005s
I20240507 08:06:45.717938 10335 raft_consensus.cc:1226] T 0c898839b7c84ce6ba0e135f778ad72d P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 9. Current term is 10. Ops: []
I20240507 08:06:45.718554 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2200) took 1667 ms. Trace:
I20240507 08:06:45.718858 10335 rpcz_store.cc:276] 0507 08:06:44.051590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.717156 (+1665566us) service_pool.cc:225] Handling call
0507 08:06:45.717645 (+ 489us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:45.718309 (+ 664us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.718485 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:45.720263 10335 raft_consensus.cc:1261] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Refusing update from remote peer 59382e6647c549d89d83c70233c0667e: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20240507 08:06:45.720594 13628 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:45.720774 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39714 (request call id 2201) took 1668 ms. Trace:
I20240507 08:06:45.676085 10192 rpcz_store.cc:276] 0507 08:06:44.161846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.246283 (+ 84437us) service_pool.cc:225] Handling call
0507 08:06:44.246638 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:44.317855 (+ 71217us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.317883 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:44.318322 (+ 439us) log.cc:842] Serialized 26 byte log entry
0507 08:06:44.900203 (+581881us) spinlock_profiling.cc:243] Waited 45.4 ms on lock 0x7b18001af680. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:06:45.082798 (+182595us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:45.082869 (+ 71us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.647547 (+564678us) spinlock_profiling.cc:243] Waited 57.6 ms on lock 0x7b540005f854. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:45.647611 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:45.647674 (+ 63us) raft_consensus.cc:1671] finished
0507 08:06:45.647687 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:45.647823 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13158,"raft.queue_time_us":16312,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":43110,"spinlock_wait_cycles":237025920,"thread_start_us":163655,"threads_started":1,"wal-append.queue_time_us":997527}
W20240507 08:06:45.222967 13115 consensus_meta.cc:287] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540: Time spent flushing consensus metadata: real 1.269s user 0.012s sys 0.004s
I20240507 08:06:45.464834 13589 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.723408 10335 rpcz_store.cc:276] 0507 08:06:44.051917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.719539 (+1667622us) service_pool.cc:225] Handling call
0507 08:06:45.719941 (+ 402us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:45.720551 (+ 610us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.720729 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:45.549633 13364 consensus_meta.cc:287] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488: Time spent flushing consensus metadata: real 1.388s user 0.011s sys 0.004s
I20240507 08:06:45.221524 13612 raft_consensus.cc:482] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.647500 13638 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.549728 13630 raft_consensus.cc:482] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.204802 13488 leader_election.cc:288] T beb5387c55024337b72bb8aa1ba09a7b P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
W20240507 08:06:45.463908 13584 log.cc:924] Time spent T 1cd78797b4574a049447c358499ad846 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 1.426s user 0.004s sys 0.008s
W20240507 08:06:45.692114 13596 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.000s sys 0.015s
W20240507 08:06:45.745226 13648 log.cc:924] Time spent T 765054c26d4a4181b4fb31891d033bb1 P bcf08c8276ba4a9ba646ad0f7090fcd8: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240507 08:06:45.745498 13086 leader_election.cc:288] T 1cd78797b4574a049447c358499ad846 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:45.727634 10259 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2054) took 1272 ms. Trace:
I20240507 08:06:45.323966 13276 raft_consensus.cc:482] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.548803 13636 raft_consensus.cc:482] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.770099 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c898839b7c84ce6ba0e135f778ad72d" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:45.763860 10940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb5387c55024337b72bb8aa1ba09a7b" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:45.438364 13606 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.502864 13643 raft_consensus.cc:482] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.770793 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:45.502966 13644 raft_consensus.cc:482] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.645489 13618 consensus_queue.cc:1046] T 63ae3c4ecd5e46ec9b5cf741db56c101 P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.248s
I20240507 08:06:45.722865 10337 consensus_queue.cc:260] T 67fb9faf0618484ab4dd4cfb90e69c28 P 88a26dac37fa4243b8c2b3d47f403488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.736471 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa60df83ab754c7e9229c7775883d22d" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:45.776599 10155 leader_election.cc:302] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59382e6647c549d89d83c70233c0667e; no voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:45.203938 13607 consensus_queue.cc:1035] T 5a168a24022a4678bd74fa028a765700 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.659s
I20240507 08:06:45.462150 13441 raft_consensus.cc:504] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.795078 10259 rpcz_store.cc:276] 0507 08:06:44.450627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.680126 (+229499us) service_pool.cc:225] Handling call
0507 08:06:44.680533 (+ 407us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:44.680886 (+ 353us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.680925 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:44.681392 (+ 467us) log.cc:842] Serialized 26 byte log entry
0507 08:06:45.315380 (+633988us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:45.339576 (+ 24196us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.339671 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:45.722724 (+383053us) raft_consensus.cc:1671] finished
0507 08:06:45.722746 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:45.722881 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":245120,"raft.queue_time_us":77978,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":14231,"thread_start_us":458438,"threads_started":1,"wal-append.queue_time_us":716658}
I20240507 08:06:45.220525 13632 raft_consensus.cc:3032] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Stepping down as leader of term 8
W20240507 08:06:46.323781 10158 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.950s user 0.005s sys 0.011s
W20240507 08:06:46.366025 10158 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.001s user 0.006s sys 0.012s
I20240507 08:06:45.363246 13532 consensus_queue.cc:1225] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Peer bcf08c8276ba4a9ba646ad0f7090fcd8 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240507 08:06:45.962495 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 453) took 1248 ms. Trace:
I20240507 08:06:46.087335 10261 raft_consensus.cc:2454] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 9.
I20240507 08:06:46.366885 10335 rpcz_store.cc:276] 0507 08:06:44.714459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.748609 (+1034150us) service_pool.cc:225] Handling call
0507 08:06:45.961831 (+213222us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:45.962110 (+ 279us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:45.962136 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:45.962215 (+ 79us) raft_consensus.cc:1631] Marking committed up to 10
0507 08:06:45.962245 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.962301 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:45.962423 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:45.777056 13375 raft_consensus.cc:482] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:45.218932 13487 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:46.226235 10263 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.000s sys 0.013s
W20240507 08:06:46.388362 10263 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s user 0.002s sys 0.015s
W20240507 08:06:46.389015 10261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 2073) took 1467 ms (client timeout 1558 ms). Trace:
W20240507 08:06:46.260946 10341 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.009s sys 0.008s
W20240507 08:06:46.389838 10341 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.634s user 0.010s sys 0.011s
W20240507 08:06:46.389559 10261 rpcz_store.cc:269] 0507 08:06:44.901120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.100745 (+199625us) service_pool.cc:225] Handling call
0507 08:06:46.367997 (+1267252us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:45.864277 10266 raft_consensus.cc:2454] T 0c898839b7c84ce6ba0e135f778ad72d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 9.
W20240507 08:06:46.326431 13651 log.cc:924] Time spent T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
W20240507 08:06:45.864732 13654 log.cc:924] Time spent T ed7ce583c04c48ebb2e80d2e0e872215 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.095s user 0.000s sys 0.002s
W20240507 08:06:46.392248 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 336452us
W20240507 08:06:45.795943 13596 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s user 0.000s sys 0.015s
I20240507 08:06:46.392665 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64f8f2cf89ed4b99822bc4389d7a842f" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:46.395488 10909 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.007s sys 0.019s
I20240507 08:06:45.783689 13562 leader_election.cc:288] T 64632a3e07184039950f3ef7fe699cb3 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:45.777040 13414 raft_consensus.cc:482] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:46.027745 13455 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.016s sys 0.054s
I20240507 08:06:46.419407 13562 raft_consensus.cc:482] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.419728 13414 raft_consensus.cc:504] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.420285 13562 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.421129 10334 raft_consensus.cc:1226] T 1f4b60e5e6294639a468641f215329a5 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:46.421828 10334 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 452) took 2690 ms. Trace:
I20240507 08:06:46.033996 13310 raft_consensus.cc:482] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.422112 10334 rpcz_store.cc:276] 0507 08:06:43.731844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.326445 (+1594601us) service_pool.cc:225] Handling call
0507 08:06:46.420590 (+1094145us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b58002c0a98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d7e050 00007fcf95d69599 00007fcf95d69042 00007fcf8f44b694
0507 08:06:46.420853 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:46.421558 (+ 705us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:46.421773 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26506624}
W20240507 08:06:46.422672 10155 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.130:41081, user_credentials={real_user=slave}} blocked reactor thread for 646526us
I20240507 08:06:45.965314 13656 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.423476 13656 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.727212 13625 consensus_queue.cc:1046] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.698s
I20240507 08:06:46.425943 13608 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:46.426077 13625 raft_consensus.cc:3032] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 8 LEADER]: Stepping down as leader of term 8
I20240507 08:06:46.302229 13570 raft_consensus.cc:482] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:46.426563 13443 raft_consensus.cc:3037] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Advancing to term 6
I20240507 08:06:46.427546 10231 leader_election.cc:302] T 7dc98f346bff4d428582b1ad1dc0fc93 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488, bcf08c8276ba4a9ba646ad0f7090fcd8; no voters:
I20240507 08:06:46.427249 13570 raft_consensus.cc:504] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.430095 13570 leader_election.cc:288] T d3f101fe713f47b1b84bcaeb54f7aead P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:46.436441 13513 raft_consensus.cc:3037] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:45.763064 13524 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.384253 13617 raft_consensus.cc:482] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.457544 13617 raft_consensus.cc:504] T 5d0e1a7b6db34cea8e152b755b52aa96 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.460886 13663 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.450407 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1765) took 1515 ms. Trace:
W20240507 08:06:46.402093 10909 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.673s user 0.007s sys 0.019s
I20240507 08:06:45.770632 13637 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.780419 13361 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.134528 13522 raft_consensus.cc:482] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:46.403517 13422 raft_consensus.cc:482] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:46.403728 13375 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:46.403565 10157 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.004s sys 0.013s
W20240507 08:06:46.472457 10157 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.759s user 0.005s sys 0.013s
W20240507 08:06:46.404937 13454 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.004s sys 0.013s
I20240507 08:06:46.006305 13442 raft_consensus.cc:482] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.142450 13548 raft_consensus.cc:482] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.405103 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:45.730286 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ac20a6d50a4700a577dfc18d71ceb2" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:46.407130 13377 raft_consensus.cc:3037] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 7 FOLLOWER]: Advancing to term 8
I20240507 08:06:45.895731 13346 raft_consensus.cc:482] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:45.894170 13636 raft_consensus.cc:504] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.005883 13589 raft_consensus.cc:504] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.514133 13672 raft_consensus.cc:2726] T 67fb9faf0618484ab4dd4cfb90e69c28 P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20240507 08:06:46.514930 13672 raft_consensus.cc:482] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:46.409873 10266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a168a24022a4678bd74fa028a765700" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
I20240507 08:06:46.409802 13487 raft_consensus.cc:504] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.411367 13492 raft_consensus.cc:3037] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Advancing to term 10
I20240507 08:06:45.777215 13628 raft_consensus.cc:482] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:45.763281 13121 raft_consensus.cc:482] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.415963 10191 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50108 (request call id 429) took 2722 ms. Trace:
I20240507 08:06:46.416445 10335 raft_consensus.cc:1226] T 8f7d05e73be847e99048092c9570faed P 88a26dac37fa4243b8c2b3d47f403488 [term 8 LEADER]: Rejecting Update request from peer 5d6656a616b747d393ff3de5328d2540 for earlier term 7. Current term is 8. Ops: []
I20240507 08:06:45.773005 13115 raft_consensus.cc:504] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.365545 13632 raft_consensus.cc:729] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 59382e6647c549d89d83c70233c0667e, State: Running, Role: LEADER
I20240507 08:06:46.136752 13638 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.418121 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "971f59883ca347648664bc7f2fcaf773" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:46.418277 13658 log.cc:924] Time spent T 64632a3e07184039950f3ef7fe699cb3 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:06:45.735944 13569 raft_consensus.cc:504] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:45.776799 13590 raft_consensus.cc:482] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
W20240507 08:06:46.419590 13455 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.224s user 0.021s sys 0.064s
I20240507 08:06:46.422508 13310 raft_consensus.cc:504] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:46.289527 13615 log.cc:924] Time spent T e0a774a8075d4c689b047bb943c78c50 P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 1.117s user 0.008s sys 0.006s
I20240507 08:06:45.778057 13642 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:46.437521 10261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ae706e85b0c447d98868f95b5a76a46" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:46.018399 13633 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.032s sys 0.064s
W20240507 08:06:46.400816 10265 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.000s sys 0.017s
I20240507 08:06:45.872726 13631 leader_election.cc:288] T 5ae706e85b0c447d98868f95b5a76a46 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.295048 13564 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.541721 10934 raft_consensus.cc:1226] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Rejecting Update request from peer 59382e6647c549d89d83c70233c0667e for earlier term 8. Current term is 9. Ops: []
I20240507 08:06:46.542136 10184 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55820 (request call id 2314) took 2387 ms. Trace:
I20240507 08:06:46.541879 13564 raft_consensus.cc:504] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.539283 13519 raft_consensus.cc:482] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:46.543017 13519 raft_consensus.cc:504] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.544128 13564 leader_election.cc:288] T db7c60767d74458094c1d86a49e87980 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:46.544659 13564 raft_consensus.cc:482] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.454578 13532 consensus_queue.cc:1035] T 971f59883ca347648664bc7f2fcaf773 P 59382e6647c549d89d83c70233c0667e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240507 08:06:46.455199 13562 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.545724 13670 raft_consensus.cc:482] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:46.545847 13562 raft_consensus.cc:482] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 88a26dac37fa4243b8c2b3d47f403488)
I20240507 08:06:46.050088 13533 raft_consensus.cc:504] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.546506 13562 raft_consensus.cc:504] T e0a774a8075d4c689b047bb943c78c50 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.134413 13630 raft_consensus.cc:504] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.464668 10340 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:39710 (request call id 1769) took 1383 ms. Trace:
I20240507 08:06:46.550673 10340 rpcz_store.cc:276] 0507 08:06:45.070147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.148772 (+ 78625us) service_pool.cc:225] Handling call
0507 08:06:46.453688 (+1304916us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:46.459244 13418 consensus_meta.cc:287] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.752s user 0.009s sys 0.004s
I20240507 08:06:46.422055 13414 leader_election.cc:288] T 316d26400fa14c4b9e67f139ffeca8f1 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.462424 10332 rpcz_store.cc:276] 0507 08:06:44.921500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:46.433806 (+1512306us) service_pool.cc:225] Handling call
0507 08:06:46.434266 (+ 460us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:46.437000 (+ 2734us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:46.437038 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0507 08:06:46.437086 (+ 48us) raft_consensus.cc:1631] Marking committed up to 5
0507 08:06:46.437113 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:46.437187 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:46.437280 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2285,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":11204}
I20240507 08:06:45.755447 13614 consensus_queue.cc:1046] T 534af8458dc14ec49ca7308cf02b685c P bcf08c8276ba4a9ba646ad0f7090fcd8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.698s
I20240507 08:06:46.426643 13625 raft_consensus.cc:729] T 5d0e1a7b6db34cea8e152b755b52aa96 P 5d6656a616b747d393ff3de5328d2540 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5d6656a616b747d393ff3de5328d2540, State: Running, Role: LEADER
I20240507 08:06:46.444916 13620 leader_election.cc:288] T 324b71380c6c47bbb335661d299baea0 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:46.467025 13490 raft_consensus.cc:482] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.540938 10936 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43936 (request call id 257) took 1599 ms. Trace:
I20240507 08:06:46.564783 10941 raft_consensus.cc:1261] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Refusing update from remote peer 88a26dac37fa4243b8c2b3d47f403488: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20240507 08:06:46.565652 10332 raft_consensus.cc:1261] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Refusing update from remote peer bcf08c8276ba4a9ba646ad0f7090fcd8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240507 08:06:46.566427 10332 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:39710 (request call id 1766) took 1644 ms. Trace:
I20240507 08:06:46.566794 10332 rpcz_store.cc:276] 0507 08:06:44.921857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:46.554474 (+1632617us) service_pool.cc:225] Handling call
0507 08:06:46.565154 (+ 10680us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:46.566121 (+ 967us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:46.566347 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:45.894436 13276 raft_consensus.cc:504] T 2a210876eb5b4a7ab6548839daccfe84 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.511130 10260 raft_consensus.cc:2454] T 64f8f2cf89ed4b99822bc4389d7a842f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 11.
W20240507 08:06:46.493392 13454 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.757s user 0.004s sys 0.013s
I20240507 08:06:46.512518 13522 raft_consensus.cc:504] T a34da9500ed541909dbdbcc3be70a067 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:46.553191 13326 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.010s sys 0.017s
I20240507 08:06:46.513164 13441 leader_election.cc:288] T f0778bf2d68148c495412379f7818b28 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:45.895018 13606 raft_consensus.cc:504] T 85b6fa5360054f32af996ec68d175584 P 88a26dac37fa4243b8c2b3d47f403488 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.516333 10267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42924 (request call id 2055) took 2065 ms. Trace:
I20240507 08:06:46.515767 13442 raft_consensus.cc:504] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.518723 13548 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.471618 13643 raft_consensus.cc:504] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } }
I20240507 08:06:46.564776 13157 raft_consensus.cc:482] T 324b71380c6c47bbb335661d299baea0 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.524534 10307 leader_election.cc:302] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88a26dac37fa4243b8c2b3d47f403488; no voters: 5d6656a616b747d393ff3de5328d2540, bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:46.527839 13628 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.528602 10263 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42914 (request call id 2071) took 2183 ms. Trace:
I20240507 08:06:46.526996 13121 raft_consensus.cc:504] T 1cd78797b4574a049447c358499ad846 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.466830 13637 raft_consensus.cc:504] T e0f47233c3be4fe99e097ade2aaec1e8 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.533197 10335 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47396 (request call id 454) took 1809 ms. Trace:
I20240507 08:06:46.533720 13375 leader_election.cc:288] T 85b6fa5360054f32af996ec68d175584 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:46.533819 13668 raft_consensus.cc:482] T db7c60767d74458094c1d86a49e87980 P 5d6656a616b747d393ff3de5328d2540 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.520915 10191 rpcz_store.cc:276] 0507 08:06:43.693364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:43.783677 (+ 90313us) service_pool.cc:225] Handling call
0507 08:06:43.804196 (+ 20519us) raft_consensus.cc:1457] Updating replica for 3 ops
0507 08:06:43.804556 (+ 360us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:43.804587 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0507 08:06:43.825037 (+ 20450us) log.cc:842] Serialized 78 byte log entry
0507 08:06:44.909859 (+1084822us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800339fa0. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf96d9e661 00007fcf96dab3e1 00007fcf8dedeeb5 00007fcf96d18939 00007fcf8defc42b 00007fcf8defa82f 00007fcf8defc992 00007fcf90aa84eb 00007fcf90aadf93 00007fcf90aae341 00007fcf90dfb81f
0507 08:06:45.206749 (+296890us) raft_consensus.cc:1631] Marking committed up to 6
0507 08:06:45.651021 (+444272us) log.cc:842] Serialized 14 byte log entry
0507 08:06:45.689522 (+ 38501us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.737287 (+ 47765us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:46.415475 (+678188us) raft_consensus.cc:1671] finished
0507 08:06:46.415599 (+ 124us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:46.415762 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32767,"prepare.queue_time_us":140962,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":55606,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":20162,"spinlock_wait_cycles":27660032,"thread_start_us":446796,"threads_started":2,"wal-append.queue_time_us":1295962}
I20240507 08:06:46.532907 13422 raft_consensus.cc:504] T 7e6f917a0bed4f6289c1abc13f6260b9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.536639 10259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9800706282e543be9f2ee3473da2b89f" candidate_uuid: "59382e6647c549d89d83c70233c0667e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" is_pre_election: true
W20240507 08:06:46.532280 10232 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.839s user 0.001s sys 0.013s
W20240507 08:06:46.538347 10265 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s user 0.000s sys 0.018s
I20240507 08:06:46.534456 13346 raft_consensus.cc:504] T 7a123878203f45319def3c4246b9fe30 P 59382e6647c549d89d83c70233c0667e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.515486 13672 raft_consensus.cc:504] T 6e83f7f3abec42d2bc76c5648cfd0aa3 P 59382e6647c549d89d83c70233c0667e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.539230 13487 leader_election.cc:288] T 7b89492a3abf415da273cc9a5b1227bf P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:46.539351 13633 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.142s user 0.033s sys 0.068s
I20240507 08:06:46.536226 13455 raft_consensus.cc:686] T 19878aa675724839ac9bb59333dee2d9 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 LEADER]: Becoming Leader. State: Replica: 88a26dac37fa4243b8c2b3d47f403488, State: Running, Role: LEADER
I20240507 08:06:46.461512 13663 raft_consensus.cc:504] T ed7ce583c04c48ebb2e80d2e0e872215 P 59382e6647c549d89d83c70233c0667e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.539302 10261 raft_consensus.cc:2379] T 5ae706e85b0c447d98868f95b5a76a46 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59382e6647c549d89d83c70233c0667e in current term 8: Already voted for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in this term.
I20240507 08:06:46.540501 13310 leader_election.cc:288] T 4cdcc48b264f4a9eb3eabce20e5bbd6c P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.426625 13608 raft_consensus.cc:504] T ffd4c65b5ad04f48916c3e359f7e45c9 P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:45.895345 13488 raft_consensus.cc:482] T ad8dd18470a542d3bec18f13b451878d P 59382e6647c549d89d83c70233c0667e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.545269 13569 leader_election.cc:288] T 3490f6d2f719474bb4f72a5647a9a726 P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 11 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.545233 13564 raft_consensus.cc:504] T 19878aa675724839ac9bb59333dee2d9 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.092072 13612 raft_consensus.cc:504] T e57f967aaaf34bd5acd85d28edeb4abe P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.546881 10185 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55816 (request call id 1911) took 1667 ms. Trace:
I20240507 08:06:46.542490 13361 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.529763 13364 raft_consensus.cc:482] T e9adf1535c974b2abf910f3c1c537627 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
W20240507 08:06:46.551241 13423 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.012s sys 0.010s
I20240507 08:06:46.562417 10942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7d05e73be847e99048092c9570faed" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:46.542496 10184 rpcz_store.cc:276] 0507 08:06:44.154647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.154785 (+ 138us) service_pool.cc:225] Handling call
0507 08:06:44.155177 (+ 392us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:44.155591 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.155631 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:44.167373 (+ 11742us) log.cc:842] Serialized 26 byte log entry
0507 08:06:44.420938 (+253565us) raft_consensus.cc:1631] Marking committed up to 2
0507 08:06:44.420991 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:44.421079 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:45.548641 (+1127562us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b44000232cc. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8de11685 00007fcf8de120c2 00007fcf90aba441 00007fcf90ab5d4a 00007fcf90aa9b1e 00007fcf90aa8e66 00007fcf90aa8a88 00007fcf90ab2481 00007fcf90ab2439 00007fcf90ab23c9
0507 08:06:46.540870 (+992229us) raft_consensus.cc:1671] finished
0507 08:06:46.540893 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:46.541942 (+ 1049us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":487,"raft.queue_time_us":23842,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":29313,"spinlock_wait_cycles":5438208,"thread_start_us":228154,"threads_started":1,"wal-append.queue_time_us":547531}
I20240507 08:06:46.535348 13524 raft_consensus.cc:504] T ad8dd18470a542d3bec18f13b451878d P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
W20240507 08:06:46.581842 10155 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.8.213.130:41081 timed out after 1.558s (SENT) [suppressed 8 similar messages]
I20240507 08:06:46.515710 13589 leader_election.cc:288] T 292ec14f14384ba1bc9ee81980e1eb2a P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:46.524335 10305 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.025s sys 0.008s
I20240507 08:06:46.548656 13632 consensus_queue.cc:260] T 19878aa675724839ac9bb59333dee2d9 P 59382e6647c549d89d83c70233c0667e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.556650 13656 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081)
I20240507 08:06:46.589303 10267 rpcz_store.cc:276] 0507 08:06:44.450935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.898682 (+447747us) service_pool.cc:225] Handling call
0507 08:06:44.899398 (+ 716us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:44.900137 (+ 739us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.900163 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:44.900738 (+ 575us) log.cc:842] Serialized 26 byte log entry
0507 08:06:45.058764 (+158026us) raft_consensus.cc:1631] Marking committed up to 3
0507 08:06:45.058796 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.065799 (+ 7003us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:46.516133 (+1450334us) raft_consensus.cc:1671] finished
0507 08:06:46.516151 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:46.516278 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":198242,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":283,"thread_start_us":157567,"threads_started":1,"wal-append.queue_time_us":503809}
I20240507 08:06:46.569052 13414 raft_consensus.cc:482] T 7a181f4ae4d342129ad262e5545debc9 P 59382e6647c549d89d83c70233c0667e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:46.546407 13670 raft_consensus.cc:504] T 1f4b60e5e6294639a468641f215329a5 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:46.569597 13590 raft_consensus.cc:504] T aa60df83ab754c7e9229c7775883d22d P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.493088 13644 raft_consensus.cc:504] T 5a168a24022a4678bd74fa028a765700 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
W20240507 08:06:47.329187 10261 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 2081) took 1567 ms (client timeout 1888 ms). Trace:
W20240507 08:06:47.329888 10261 rpcz_store.cc:269] 0507 08:06:45.762038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:46.400152 (+638114us) service_pool.cc:225] Handling call
0507 08:06:47.329127 (+928975us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:46.774960 10340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488" is_pre_election: true
I20240507 08:06:46.602684 10260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534af8458dc14ec49ca7308cf02b685c" candidate_uuid: "88a26dac37fa4243b8c2b3d47f403488" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8"
I20240507 08:06:47.039255 10934 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43936 (request call id 255) took 2104 ms. Trace:
I20240507 08:06:47.331154 10335 rpcz_store.cc:276] 0507 08:06:44.714758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:46.415681 (+1700923us) service_pool.cc:225] Handling call
0507 08:06:46.416116 (+ 435us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:46.523586 (+107470us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:46.524026 (+ 440us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:06:47.328413 10156 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.131:37047, user_credentials={real_user=slave}} blocked reactor thread for 865082us
W20240507 08:06:47.048593 10307 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.8.213.132:44849, user_credentials={real_user=slave}} blocked reactor thread for 580355us
W20240507 08:06:46.586867 13326 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.813s user 0.010s sys 0.018s
I20240507 08:06:46.761761 10333 raft_consensus.cc:2454] T 5ae706e85b0c447d98868f95b5a76a46 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 8.
W20240507 08:06:47.346333 10305 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.380s user 0.029s sys 0.015s
W20240507 08:06:47.347018 10232 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.841s user 0.001s sys 0.020s
W20240507 08:06:47.355355 10232 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.8.213.129:32795, user_credentials={real_user=slave}} blocked reactor thread for 1.85074e+06us
I20240507 08:06:46.726559 10258 raft_consensus.cc:2454] T 2e5fd8588b9240c38e1f3c75cf342156 P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88a26dac37fa4243b8c2b3d47f403488 in term 8.
W20240507 08:06:46.983742 10138 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 82c5eb4243284619ad55e8a724ff32de, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:47.330462 10185 rpcz_store.cc:276] 0507 08:06:44.879210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.887752 (+ 8542us) service_pool.cc:225] Handling call
0507 08:06:44.888120 (+ 368us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:44.888695 (+ 575us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:44.888740 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:44.889390 (+ 650us) log.cc:842] Serialized 26 byte log entry
0507 08:06:45.107958 (+218568us) raft_consensus.cc:1631] Marking committed up to 4
0507 08:06:45.108008 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:45.108117 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:46.546671 (+1438554us) raft_consensus.cc:1671] finished
0507 08:06:46.546697 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:46.546823 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34642,"raft.queue_time_us":197687,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":185,"thread_start_us":31994,"threads_started":1,"wal-append.queue_time_us":756998}
I20240507 08:06:47.331498 10340 raft_consensus.cc:2454] T e940ba6acb8d4782a4c921b50b882f91 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf08c8276ba4a9ba646ad0f7090fcd8 in term 7.
I20240507 08:06:46.580194 13519 leader_election.cc:288] T 50d914d033cf48c389262dd8b7c7fbce P 88a26dac37fa4243b8c2b3d47f403488 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 5d6656a616b747d393ff3de5328d2540 (127.8.213.132:44849)
I20240507 08:06:46.757081 10339 raft_consensus.cc:595] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Received request to transfer leadership to TS bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:47.329526 10259 raft_consensus.cc:2454] T 9800706282e543be9f2ee3473da2b89f P bcf08c8276ba4a9ba646ad0f7090fcd8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59382e6647c549d89d83c70233c0667e in term 10.
I20240507 08:06:47.360869 10339 raft_consensus.cc:603] T 316d26400fa14c4b9e67f139ffeca8f1 P 88a26dac37fa4243b8c2b3d47f403488 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20240507 08:06:47.041314 13674 log.cc:924] Time spent T 82c5eb4243284619ad55e8a724ff32de P 59382e6647c549d89d83c70233c0667e: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
W20240507 08:06:46.762279 13667 log.cc:924] Time spent T 7dc98f346bff4d428582b1ad1dc0fc93 P 88a26dac37fa4243b8c2b3d47f403488: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20240507 08:06:47.328512 10263 rpcz_store.cc:276] 0507 08:06:44.340626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:44.423617 (+ 82991us) service_pool.cc:225] Handling call
0507 08:06:44.514727 (+ 91110us) raft_consensus.cc:1457] Updating replica for 1 ops
0507 08:06:45.703146 (+1188419us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:06:45.703193 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0507 08:06:45.703790 (+ 597us) log.cc:842] Serialized 26 byte log entry
0507 08:06:46.389018 (+685228us) raft_consensus.cc:1631] Marking committed up to 7
0507 08:06:46.389073 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:46.394966 (+ 5893us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0507 08:06:46.516074 (+121108us) raft_consensus.cc:1671] finished
0507 08:06:46.516098 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0507 08:06:46.523511 (+ 7413us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":159561,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":109,"thread_start_us":1871766,"threads_started":2,"wal-append.queue_time_us":808943}
W20240507 08:06:47.361320 10259 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42914 (request call id 2082) took 1470 ms (client timeout 1807 ms). Trace:
W20240507 08:06:47.367735 10259 rpcz_store.cc:269] 0507 08:06:45.891212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:46.526270 (+635058us) service_pool.cc:225] Handling call
0507 08:06:47.361243 (+834973us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:47.332512 10934 rpcz_store.cc:276] 0507 08:06:44.935055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.070497 (+135442us) service_pool.cc:225] Handling call
0507 08:06:45.098029 (+ 27532us) raft_consensus.cc:1457] Updating replica for 0 ops
0507 08:06:46.542181 (+1444152us) raft_consensus.cc:1683] Filling consensus response to leader.
0507 08:06:47.038923 (+496742us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400730fd4. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf90dac6b1 00007fcf90e281a1 00007fcf90e26bee 00007fcf95d69089 00007fcf8f44b694 00007fcf8f44b622 00007fcf8f44b551 00007fcf8f44b4cc 00007fcf8f44a782 00007fcf8f24ebcc 00007fcf8f24e006
0507 08:06:47.039172 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3287346944}
W20240507 08:06:47.330369 13423 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.601s user 0.017s sys 0.025s
I20240507 08:06:46.754288 13115 leader_election.cc:288] T 03d2e621edea41ae97e79e36e4dc0365 P 5d6656a616b747d393ff3de5328d2540 [CANDIDATE]: Term 9 election: Requested vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
W20240507 08:06:47.363204 10260 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42924 (request call id 2062) took 1723 ms (client timeout 2089 ms). Trace:
I20240507 08:06:46.602697 10941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940ba6acb8d4782a4c921b50b882f91" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5d6656a616b747d393ff3de5328d2540" is_pre_election: true
I20240507 08:06:47.142040 13442 leader_election.cc:288] T cd9cd8b1faef46a9bbbee733353fd123 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.601949 13620 raft_consensus.cc:482] T 8933e79c6c78466498a6d17d93ce7cff P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:47.240674 13492 consensus_meta.cc:287] T bfb70b9953fd486ca5f88b5999dd26a0 P 59382e6647c549d89d83c70233c0667e: Time spent flushing consensus metadata: real 0.717s user 0.009s sys 0.000s
W20240507 08:06:47.356405 10138 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 82c5eb4243284619ad55e8a724ff32de, TS 59382e6647c549d89d83c70233c0667e -> TS 5d6656a616b747d393ff3de5328d2540 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "88a26dac37fa4243b8c2b3d47f403488" member_type: VOTER last_known_addr { host: "127.8.213.131" port: 37047 } }
I20240507 08:06:46.945714 13677 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 88a26dac37fa4243b8c2b3d47f403488 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:47.041014 13513 raft_consensus.cc:2726] T d3f101fe713f47b1b84bcaeb54f7aead P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20240507 08:06:47.357261 10156 leader_election.cc:302] T 10eece2dd4a64e3aa99d91fadc43df89 P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59382e6647c549d89d83c70233c0667e, 88a26dac37fa4243b8c2b3d47f403488; no voters: bcf08c8276ba4a9ba646ad0f7090fcd8
I20240507 08:06:47.235906 13673 raft_consensus.cc:482] T 7b89492a3abf415da273cc9a5b1227bf P 88a26dac37fa4243b8c2b3d47f403488 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:46.861645 13454 raft_consensus.cc:482] T ffd4c65b5ad04f48916c3e359f7e45c9 P 5d6656a616b747d393ff3de5328d2540 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:47.363273 13532 raft_consensus.cc:482] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:47.373154 13487 raft_consensus.cc:482] T 84da12b367b046b7bcfb08aaa6e1c97b P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5d6656a616b747d393ff3de5328d2540)
I20240507 08:06:46.759003 13640 raft_consensus.cc:482] T 8f7d05e73be847e99048092c9570faed P 5d6656a616b747d393ff3de5328d2540 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:06:47.373382 10260 rpcz_store.cc:269] 0507 08:06:45.639968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:46.588174 (+948206us) service_pool.cc:225] Handling call
0507 08:06:47.363140 (+774966us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:06:46.585119 13490 raft_consensus.cc:504] T c6ac20a6d50a4700a577dfc18d71ceb2 P 5d6656a616b747d393ff3de5328d2540 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129" port: 32795 } } peers { permanent_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" member_type: VOTER last_known_addr { host: "127.8.213.130" port: 41081 } } peers { permanent_uuid: "5d6656a616b747d393ff3de5328d2540" member_type: VOTER last_known_addr { host: "127.8.213.132" port: 44849 } attrs { promote: false } }
I20240507 08:06:47.275135 13660 raft_consensus.cc:482] T 03d2e621edea41ae97e79e36e4dc0365 P 88a26dac37fa4243b8c2b3d47f403488 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 59382e6647c549d89d83c70233c0667e)
I20240507 08:06:46.596242 13533 leader_election.cc:288] T 95da5fd11446429481eb60f5a0c8158e P 59382e6647c549d89d83c70233c0667e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers bcf08c8276ba4a9ba646ad0f7090fcd8 (127.8.213.130:41081), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:47.275377 13332 raft_consensus.cc:482] T bfb70b9953fd486ca5f88b5999dd26a0 P 88a26dac37fa4243b8c2b3d47f403488 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:06:47.367086 10335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3490f6d2f719474bb4f72a5647a9a726" candidate_uuid: "bcf08c8276ba4a9ba646ad0f7090fcd8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "88a26dac37fa4243b8c2b3d47f403488"
I20240507 08:06:46.755210 13636 leader_election.cc:288] T b909813a654d41cf88fb9fee2f8006bc P bcf08c8276ba4a9ba646ad0f7090fcd8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 59382e6647c549d89d83c70233c0667e (127.8.213.129:32795), 88a26dac37fa4243b8c2b3d47f403488 (127.8.213.131:37047)
I20240507 08:06:46.569787 10936 rpcz_store.cc:276] 0507 08:06:44.935852 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:06:45.073001 (+137149us) service_pool.cc:225] Handling call
0507 08:06:45.729964 (+656963us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x7b58002f2e98. stack: 00007fcf8decaba3 00007fcf8decaaa0 00007fcf8d82f584 00007fcf975389d4 00007fcf97538969 00007fcf8dd43b51 00007fcf8dec53b1 00007fcf8dec5262 00007fcf96db7889 00007fcf95cf43c0 00007fcf95dba3ab 00007fcf95dba431 00007fcf95d69f50 00007fcf95d69834 00007fcf8f44dc14 00007fcf8f44dba2
0507 08:06:46.535269 (+805305us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3645312}
I20240507 08:06:47.423964 13532 raft_consensus.cc:504] T bcbe630c1736458898143fb230a9ffb8 P 59382e6647c549d89d83c70233c0667e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "59382e6647c549d89d83c70233c0667e" member_type: VOTER last_known_addr { host: "127.8.213.129"